modelbeast/HANDOFF_HY3D_MLX.md

# HANDOFF: `hunyuan3d_mlx` operator — fully-local Hunyuan3D 2.1 via MLX

**For:** Opus 4.8 executing autonomously in this repo.
**Goal:** Add a second fully-local image→3D generator (Hunyuan3D 2.1, native MLX) alongside `trellis_mac`, benchmark it on both nodes, and (stretch) fork-improve it to exploit Studio-class GPUs (M3 Ultra 256GB / 819GB/s, M1 Ultra 128GB). Written 2026-07-16 after code-level assessment of the upstream repo.

---

## 1. Context & assessment (read first, don't re-derive)

**Upstream:** https://github.com/dgrauet/Hunyuan3D-2.1-mlx — fork of Tencent's Hunyuan3D-2.1 adding native MLX inference for BOTH stages:
- **Stage 1** (image → mesh): `hy3dshape/hy3dshape/pipeline_mlx.py` → `ShapePipeline.from_pretrained("dgrauet/hunyuan3d-2.1-mlx")`. Validated numerically vs PyTorch within 1e-5.
- **Stage 2** (mesh + reference image → PBR-textured GLB): `hy3dpaint/textureGenPipeline_mlx.py` → `Hunyuan3DPaintPipelineMLX`. UNet match 1.17e-5 vs PyTorch. Includes MLX RealESRGAN x4 (512²→2048² per view), cosine-weighted bake blend, EDT UV-gutter fill, proper glTF PBR export (`baseColorTexture` + `metallicRoughnessTexture`, doubleSided).

**Weights:** HF `dgrauet/hunyuan3d-2.1-mlx`, ~15.1GB total, auto-downloaded on first use. Override dir via env `HUNYUAN3D_MLX_WEIGHTS_DIR`. FP16 DiT is 5.68GB (`dit.safetensors`); paint stack adds `paint_unet` 3.66GB + `paint_dino` 2.12GB + `paint_clip` 1.18GB + VAEs + `realesrgan_x4plus_mlx`. Peak memory ~10GB stage 1 fp16 + ~6GB stage 2 → trivial on both our nodes. **Use FP16, not INT8** — both Ultras have memory to burn and the owner wants best quality; INT8/INT4 exist only for 16GB Macs.

**Port quality verdict: high for a 4-star repo.** Disciplined single-author work: `docs/forward_pass.md` documents porting principles, `docs/adr/` has architecture decisions, `tests/` includes an end-to-end stage1→stage2 chain and unit tests. Deviations from the PyTorch reference are documented, not silent. Rough edges to expect: single author, ~188 weight downloads, no community battle-testing — treat every first run as a probe.

**Reference timings (M2 Pro):** ~9 min for 6-view texture synthesis at 512px. Our Ultras should beat this substantially; M1 Ultra runs MLX natively (unlike TRELLIS.2's torch-MPS bf16 question flagged in AGENTS.md).

### The 4096-texture situation (this is the fork opportunity)

The README claims `texture_size=2048` vs PyTorch's 4096 because "the MLX Metal rasterizer has no tiling." **The code is ahead of the README:**

- `hy3dpaint/textureGenPipeline_mlx.py:82` — config default is ALREADY `texture_size = 1024 * 4` (4096).
- `hy3dpaint/DifferentiableRenderer/mesh_render_mlx.py:267` — `_rasterize_tiled(pos_clip, resolution, tiles=4)` exists, verified against single-dispatch at 2048².
- `mesh_render_mlx.py:647-658` — `extract_textiles()` (the main UV-space raster feeding `back_project(method="back_sample")`) already auto-tiles: 2 tiles at ≥3072, 4 tiles at ≥4096.
- **Remaining untiled gaps:** `uv_feature_map()` at `mesh_render_mlx.py:728` calls plain `_rasterize(vtx_uv_clip, self.texture_size)` — this will blow the Metal command-buffer budget at 4096 if it's on the hot path. (`mesh_render_mlx.py:815` is camera-view raster at `render_size`=2048, fine as-is.) The e2e test (`hy3dpaint/tests/test_e2e_mesh.py:49-51`) pins 2048 "because we don't tile the bake."
- The command-buffer error (`kIOGPUCommandBufferCallbackErrorImpactingInteractivity`) is a watchdog concern documented **on laptop GPUs**. An M3 Ultra Studio may simply not hit it untiled — test before patching.

Other Studio-class headroom knobs (all currently tuned for laptops):
- **Remesh budget:** `tests/test_stage1_to_stage2.py:82-91` decimates Stage 1's ~500k-face mesh to ~40-60k before texturing (rasterizer budget again). On Ultra, try 100-200k for bake fidelity.
- **Stage 1 `octree_resolution`:** default 256; try 384 for finer geometry (memory is not our constraint).
- **`max_num_view`:** default 6; the candidate camera list supports many more views.

---

## 2. Phase A — Fork + vendor + install (M3 first)

1. **Fork** `dgrauet/Hunyuan3D-2.1-mlx` to the owner's GitHub account with `gh repo fork dgrauet/Hunyuan3D-2.1-mlx --clone=false` (confirm the account `gh auth status` shows; do NOT create the fork under an org without asking). All our patches go to the fork's `modelbeast` branch; keep `main` tracking upstream for rebases.
2. **Vendor:** clone the fork to `vendor/hunyuan3d-mlx` (repo convention: `vendor/trellis-mac`, `vendor/stable-fast-3d`).
3. **Install script** `scripts/install_hunyuan3d_mlx.sh`, modeled on `scripts/install_trellis_mac.sh`:
   - Create `vendor/hunyuan3d-mlx/.venv` (python3.11, same as trellis-mac).
   - `pip install mlx mlx-arsenal safetensors Pillow trimesh scikit-image PyMCubes scipy huggingface_hub xatlas opencv-python` (README's list) **plus** `pymeshlab` (needed for `use_remesh=True` — see `textureGenPipeline_mlx.py:39-41`, it soft-fails to None without it).
   - NOTE: Stage 2's `Hunyuan3DPaintConfigMLX` references torch-era paths (`device="cuda"`, `multiview_pretrained_path`) but with `use_mlx_diffusion=True` (the default) the PyTorch multiview model is skipped — verify at runtime whether `torch` is still imported anywhere on the MLX path before adding it to the venv. Prefer NOT installing torch if the MLX path doesn't need it.
   - Verify: `.venv/bin/python -c "import mlx.core as mx; print(mx.default_device())"`.
4. **Env guards:** reuse the `HF_HUB_DISABLE_XET=1` guard from `trellis_mac/run.py` (xet downloader breaks on some repos); the OMP guards likely unneeded without torch but harmless.

## 3. Phase B — CLI wrapper + operator

The upstream repo has no single-command CLI for the full chain — `tests/test_stage1_to_stage2.py` is the reference implementation of image→textured-GLB. Build:

1. **In the fork** (so it's upstreamable): `generate_e2e.py` at repo root —
   `python generate_e2e.py input.png --output out_stem [--steps 50] [--guidance 7.5] [--octree-resolution 256] [--max-num-view 6] [--view-resolution 512] [--texture-size 2048|4096] [--seed 42] [--no-texture] [--remesh-faces 40000]`
   - Stage 1: `ShapePipeline.from_pretrained(...)`, export `<stem>_shape.glb`.
   - If `--no-texture`, stop there. Else Stage 2 per the test file: remesh → `Hunyuan3DPaintPipelineMLX` → `<stem>.glb`.
   - Print stage timings and peak memory (`mx.get_peak_memory()`) — we want these in job logs.
2. **Operator** `server/operators/hunyuan3d_mlx/` mirroring `trellis_mac` exactly:
   - `manifest.json`: id `hunyuan3d_mlx`, name "Hunyuan3D 2.1 (local MLX)", category `mesh-gen`, accepts `["image"]`, produces `["model"]`, resources `"gpu"`, python `vendor/hunyuan3d-mlx/.venv/bin/python`. Params: `steps` (default 50), `guidance_scale` (7.5), `octree_resolution` (enum 256/384, default 256), `texture_size` (enum 1024/2048/4096, default 2048 until Phase D proves 4096), `max_num_view` (default 6), `seed`, `no_texture`.
   - `run.py`: copy `trellis_mac/run.py` shape — subprocess `generate_e2e.py`, cwd vendor dir, collect `*.glb` (prefer the textured one over `*_shape.glb`), write `result.json` with `{"outputs": [...], "meta": {"tool": "hunyuan3d_mlx"}}`.
3. **Smoke test on scratch DB** (per repo rule: never test against the running gens DB): `./mb run hunyuan3d_mlx --file <test image> --wait --download out/`. First run downloads ~15GB — job log shows progress, don't kill it (AGENTS.md warns 10-30+ min).
4. Best-practice chain: `fal_bg_remove` (or local `bg_remove_local`) → cutout → `hunyuan3d_mlx`, same as the fal chain in AGENTS.md.

## 4. Phase C — M1 Ultra node + benchmarks

1. Install on the M1 node (`http://100.91.239.7:8777`, see AGENTS.md) — same scripts; MLX is architecture-native on M1, no bf16 concern. Mind the unified worker pool (commit `b8c333a`): check how the gpu lane discovers operators per node in `server/runner.py` / `server/remote.py` before assuming both nodes can serve the op.
2. **Benchmark protocol** — same cutout image through all three, one at a time (gpu lane serializes anyway):
   - `trellis_mac` (M3, defaults: 1024 pipeline, 2048 texture)
   - `hunyuan3d_mlx` on M3 AND on M1 (fp16, defaults)
   - `fal_hunyuan3d_v21` (single-image mode — multi-view is broken on fal, verified 2026-07)
   Record per run: wall time per stage, peak memory, GLB size, vert/face count, and a viewer screenshot. Subjective quality notes: geometry coherence, texture seams, PBR plausibility.
3. **Record in BENCHMARKS.md** following the existing table format, and update the local-verdict line if Hunyuan changes the recommendation (e.g. "trellis_mac for geometry, hunyuan3d_mlx for textures"). Add the operator row to README.md's table and AGENTS.md's mesh-gen row.

## 5. Phase D (stretch) — Studio-class fork improvements

Work on the fork's `modelbeast` branch; each item is independently shippable and upstreamable as a PR to dgrauet:

1. **4096² textures.** First just run `--texture-size 4096` on the M3 Ultra untiled — the command-buffer watchdog may not fire on Studio GPUs. If it fires: patch `uv_feature_map()` (`mesh_render_mlx.py:728`) to route through the existing `_rasterize_tiled()` using the same ≥3072/≥4096 tile thresholds as `extract_textiles()` (line 647). Grep for any other `_rasterize(..., self.texture_size)` call. Validate: tiled output must equal untiled at 2048 (the codebase already did this for `extract_textiles`; reuse that method). Compare 2048 vs 4096 bakes on the same mesh — if 4096 wins visibly, flip the operator default on the M3.
2. **Bigger bake mesh.** Raise the remesh decimation target from ~40k to 100-200k faces (flag already exposed via `--remesh-faces`); check bake time scaling and whether the rasterizer holds.
3. **Stage 1 detail:** `octree_resolution=384` A/B vs 256 on a detail-heavy cutout.
4. **More views:** `max_num_view=8-12` A/B — the candidate camera ring (30° azimuth steps at ±20° elevation, `textureGenPipeline_mlx.py:89-100`) already supports it. Watch for texture-blend regressions, not just wall time.
5. Log every result in BENCHMARKS.md; if any patch is a clear win, open the upstream PR (small, focused, with the validation evidence — the author clearly values PT-parity proofs).

## 6. Guardrails

- Never test against the production DB with running generations — scratch DB only (repo rule).
- gpu lane = ONE job at a time; don't parallelize benchmarks on a node.
- First-run weight downloads are long; don't cancel "stuck" jobs that are downloading.
- The reference image must match the mesh content (upstream caveat) — for e2e runs this is automatic (same image drives both stages).
- If Stage 2 quality disappoints out of the box, benchmark Stage 1 + `--no-texture` anyway: Hunyuan shape + TRELLIS texture routing is a valid future pipeline.
- Ask the owner before: forking under an org account, pushing PRs upstream, or changing existing operator defaults.