README: MODELBEAST build overview (provenance, our changes, quick start, hardware, license); upstream README -> README_UPSTREAM.md
This commit is contained in:
modelbeast
parent
e4cfa9d1e9
commit
04fd1ceb28
301
README.md
301
README.md
@ -1,282 +1,71 @@
|
||||
# Hunyuan3D-2.2-mrp-MLX
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/images/teaser.jpg">
|
||||
</p>
|
||||
|
||||
|
||||
<div align="center">
|
||||
<a href=https://3d.hunyuan.tencent.com target="_blank"><img src=https://img.shields.io/badge/Official%20Site-333399.svg?logo=homepage height=22px></a>
|
||||
<a href=https://huggingface.co/spaces/tencent/Hunyuan3D-2.1 target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20Demo-276cb4.svg height=22px></a>
|
||||
<a href=https://huggingface.co/tencent/Hunyuan3D-2.1 target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20Models-d96902.svg height=22px></a>
|
||||
<a href=https://3d-models.hunyuan.tencent.com/ target="_blank"><img src= https://img.shields.io/badge/Page-bb8a2e.svg?logo=github height=22px></a>
|
||||
<a href=https://discord.gg/dNBrdrGGMa target="_blank"><img src= https://img.shields.io/badge/Discord-white.svg?logo=discord height=22px></a>
|
||||
<a href=https://arxiv.org/pdf/2506.15442 target="_blank"><img src=https://img.shields.io/badge/Report-b5212f.svg?logo=arxiv height=22px></a>
|
||||
<a href=https://x.com/TencentHunyuan target="_blank"><img src=https://img.shields.io/badge/Hunyuan-black.svg?logo=x height=22px></a>
|
||||
<a href="#community-resources" target="_blank"><img src=https://img.shields.io/badge/Community-lavender.svg?logo=homeassistantcommunitystore height=22px></a>
|
||||
</div>
|
||||
|
||||
[//]: # ( <a href=# target="_blank"><img src=https://img.shields.io/badge/Report-b5212f.svg?logo=arxiv height=22px></a>)
|
||||
|
||||
[//]: # ( <a href=# target="_blank"><img src= https://img.shields.io/badge/Colab-8f2628.svg?logo=googlecolab height=22px></a>)
|
||||
|
||||
[//]: # ( <a href="#"><img alt="PyPI - Downloads" src="https://img.shields.io/pypi/v/mulankit?logo=pypi" height=22px></a>)
|
||||
<br>
|
||||
|
||||
## Apple MLX Port (Apple Silicon)
|
||||
|
||||
This fork adds native **MLX inference** for the full Hunyuan3D-2.1 pipeline on Apple Silicon Macs — both shape generation (image → mesh) and PBR texture synthesis (mesh + reference image → textured GLB).
|
||||
|
||||
### Weights
|
||||
|
||||
Pre-converted MLX weights are hosted on HuggingFace and auto-downloaded on first use:
|
||||
|
||||
> [`dgrauet/hunyuan3d-2.1-mlx`](https://huggingface.co/dgrauet/hunyuan3d-2.1-mlx)
|
||||
|
||||
If you'd rather convert from the original PyTorch checkpoints yourself:
|
||||
**One-command image → textured 3D on Apple Silicon.** A tuned, packaged build of Tencent's Hunyuan3D 2.1, running natively on Mac GPUs via Apple MLX — no NVIDIA, no CUDA, no cloud.
|
||||
|
||||
```bash
|
||||
pip install mlx-forge
|
||||
mlx-forge convert hunyuan3d-2.1 --output ./models/hunyuan3d-2.1-mlx
|
||||
# or quantized for 16GB Macs:
|
||||
mlx-forge convert hunyuan3d-2.1 --quantize --bits 8 --output ./models/hunyuan3d-2.1-mlx
|
||||
python generate_e2e.py your_image.png --output out/mymesh
|
||||
# → out/mymesh.glb (PBR-textured mesh) + out/mymesh_shape.glb (untextured)
|
||||
```
|
||||
|
||||
Override the default HF lookup by setting `HUNYUAN3D_MLX_WEIGHTS_DIR=/path/to/local/models`.
|
||||
|
||||
### Install
|
||||
|
||||
```bash
|
||||
pip install mlx mlx-arsenal safetensors Pillow trimesh scikit-image PyMCubes scipy
|
||||
pip install huggingface_hub xatlas opencv-python # for Stage 2
|
||||
```
|
||||
|
||||
### Stage 1 — Shape Generation (image → mesh)
|
||||
|
||||
```python
|
||||
from hy3dshape.hy3dshape.pipeline_mlx import ShapePipeline
|
||||
|
||||
pipe = ShapePipeline.from_pretrained("dgrauet/hunyuan3d-2.1-mlx")
|
||||
mesh = pipe("your_image.png", num_inference_steps=50, guidance_scale=7.5, octree_resolution=256)
|
||||
mesh.export("output.glb")
|
||||
```
|
||||
|
||||
### Stage 2 — PBR Texture Synthesis (mesh + reference → textured GLB)
|
||||
|
||||
```python
|
||||
from textureGenPipeline_mlx import Hunyuan3DPaintConfigMLX, Hunyuan3DPaintPipelineMLX
|
||||
|
||||
cfg = Hunyuan3DPaintConfigMLX(max_num_view=6, resolution=512)
|
||||
pipe = Hunyuan3DPaintPipelineMLX(cfg)
|
||||
pipe(
|
||||
mesh_path="mesh.glb",
|
||||
image_path="reference.png",
|
||||
output_mesh_path="textured.obj",
|
||||
save_glb=True,
|
||||
)
|
||||
```
|
||||
|
||||
Output: `textured.glb` with 2048² PBR albedo + metallic-roughness textures baked onto the mesh UVs (~9 min for 6 views at 512px on an M2 Pro).
|
||||
|
||||
Defaults match the PyTorch reference config exactly, with one documented exception: `texture_size=2048` (PT uses 4096). The MLX Metal rasterizer has no tiling and would exceed the GPU command-buffer budget at 4096² on laptop GPUs (`kIOGPUCommandBufferCallbackErrorImpactingInteractivity`). Every other knob — `max_num_view`, `render_size`, `num_inference_steps`, `guidance_scale`, bake mode, inpaint method — is PT-identical.
|
||||
|
||||
### End-to-end (image → textured GLB)
|
||||
|
||||
`tests/test_stage1_to_stage2.py` chains both stages on a single reference image. Stage 1's dense marching-cubes mesh is automatically remeshed to ~40k faces before Stage 2's bake (the Metal rasterizer can't handle Stage 1's raw ~500k faces at `texture_size=2048`).
|
||||
|
||||
### Memory Requirements
|
||||
|
||||
| Precision | DiT Size | Peak Memory | Recommended Mac |
|
||||
|-----------|----------|-------------|-----------------|
|
||||
| FP16 | 5.7 GB | ~10 GB | 32 GB+ |
|
||||
| INT8 | 3.0 GB | ~6 GB | 16 GB+ |
|
||||
| INT4 | 1.6 GB | ~4 GB | 16 GB |
|
||||
|
||||
Stage 2 (paint) adds ~6 GB peak for the UNet + VAE + DINOv2 at fp32.
|
||||
|
||||
### Scope
|
||||
|
||||
- ✅ **Stage 1** (shape generation): fully ported to MLX, validated numerically against PyTorch (1e-5)
|
||||
- ✅ **Stage 2** (PBR texture synthesis): fully ported to MLX, full UNet match within 1.17e-5 vs PyTorch
|
||||
- Metal rasterizer, UNet 2.5D with dual-stream reference attention, VAE, DINOv2, v_prediction scheduler
|
||||
- **Cosine-weighted bake blend** (PT-parity `fast_bake_texture`) smooths across face seams
|
||||
- **Mesh-aware vertex-color propagation + face barycentric raster**, `cv2.INPAINT_NS` (PT-parity), plus an EDT nearest-fill post-pass that pads UV gutters so 3D viewers doing bilinear sampling across island boundaries never pull in the atlas background
|
||||
- **RealESRGAN x4 super-resolution** in MLX (512² → 2048² per view before bake)
|
||||
- **GLB export**: PBRMaterial with sRGB `baseColorTexture`, `metallicRoughnessTexture` (glTF channel order), `doubleSided=true`
|
||||
- All pipeline defaults now match the PyTorch reference exactly (see porting principle in `docs/forward_pass.md`)
|
||||
|
||||
### Texture quality notes
|
||||
|
||||
- Reference image must match the mesh content (e.g. mermaid image on mermaid mesh). Cross-pairing produces fragmented atlases since the diffusion's albedo views can't coherently project onto an unrelated 3D layout.
|
||||
- Tested on `assets/case_1/mesh.glb` (fox) and `assets/case_2/mesh.glb` (mermaid) — all 6/6 views clean with PT-aligned defaults.
|
||||
|
||||
---
|
||||
|
||||
## 🔥 News
|
||||
## What this is (provenance & credit)
|
||||
|
||||
- Jul 26, 2025: 🤗 We release the first open-source, simulation-capable, immersive 3D world generation model, [HunyuanWorld-1.0](https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0)!
|
||||
- Jun 19, 2025: 👋 We present the [technical report](https://arxiv.org/pdf/2506.15442) of Hunyuan3D-2.1, please check out the details and spark some discussion!
|
||||
- Jun 13, 2025: 🤗 We release the first production-ready 3D asset generation model, Hunyuan3D-2.1!
|
||||
A **fork of a fork** — three layers, credited honestly:
|
||||
|
||||
> Join our **[Wechat](#)** and **[Discord](https://discord.gg/dNBrdrGGMa)** group to discuss and find help from us.
|
||||
1. **Tencent Hunyuan3D 2.1** — the actual models: `Hunyuan3D-Shape-v2-1` (3.3B, image→mesh) + `Hunyuan3D-Paint-v2-1` (2B, PBR texture). Originally **CUDA/NVIDIA only**.
|
||||
2. **[dgrauet/Hunyuan3D-2.1-mlx](https://github.com/dgrauet/Hunyuan3D-2.1-mlx)** — the hard part: a native **MLX (Metal)** port of *both* stages. fp16, no bf16 dependency → runs on **any** Apple Silicon Mac (M1–M5). This is what made it work on a Mac at all.
|
||||
3. **This build (MODELBEAST / `mrp`)** — the usability + tuning layer on top of dgrauet (below).
|
||||
|
||||
| Wechat Group | Xiaohongshu | X | Discord |
|
||||
|--------------------------------------------------|-------------------------------------------------------|---------------------------------------------|---------------------------------------------------|
|
||||
| <img src="assets/qrcode/wechat.png" height=140> | <img src="assets/qrcode/xiaohongshu.png" height=140> | <img src="assets/qrcode/x.png" height=140> | <img src="assets/qrcode/discord.png" height=140> |
|
||||
## What *we* added over upstream
|
||||
|
||||
## 🤗 Community Contribution Leaderboard
|
||||
1. By [@visualbruno](https://github.com/visualbruno)
|
||||
- ComfyUI-Hunyuan3d-2-1: https://github.com/visualbruno/ComfyUI-Hunyuan3d-2-1
|
||||
2. By [@VR-Jobs](https://github.com/VR-Jobs)
|
||||
- Hunyuan3d-2-1 Unity Support: https://github.com/VR-Jobs/Hunyuan3D-2.1-Unity-XR-PC-Phone
|
||||
- **`generate_e2e.py`** — a single command chaining Stage 1 (shape) → Stage 2 (PBR texture). Upstream only shipped a test script. Flags: `--steps --guidance --octree-resolution --max-num-view --view-resolution --texture-size --remesh-faces --seed --no-texture`; prints per-stage timings + peak memory.
|
||||
- **Env-tunable bake mesh** — `HY3D_REMESH_FACES` (upstream hard-coded 40k). The **single biggest quality lever** — it's what un-melts faces.
|
||||
- **Studio-quality defaults** — `octree 384 / texture 4096 / remesh 120k`, **verified watchdog-free on M3 *and* M1 Ultra**. Upstream capped texture at 2048 because laptop GPUs hit a Metal command-buffer limit; Studio-class GPUs don't, and the existing tiling handles 4096².
|
||||
- **[HARDWARE.md](HARDWARE.md)** — exact RAM needs per precision and per Mac.
|
||||
- **Slimmed for inference** — upstream training data (`mini_trainset`) + demo images stripped (~165 MB → ~9 MB). Recover any time via `git fetch upstream` (github.com/dgrauet).
|
||||
|
||||
## ☯️ **Hunyuan3D 2.1**
|
||||
|
||||
### Architecture
|
||||
|
||||
Tencent Hunyuan3D-2.1 is a scalable 3D asset creation system that advances state-of-the-art 3D generation through two pivotal innovations: Fully Open-Source Framework and Physically-Based Rendering (PBR) Texture Synthesis. For the first time, the system releases full model weights and training code, enabling community developers to directly fine-tune and extend the model for diverse downstream applications. This transparency accelerates academic research and industrial deployment. Moreover, replacing the prior RGB-based texture model, the upgraded PBR pipeline leverages physics-grounded material simulation to generate textures with photorealistic light interaction (e.g., metallic reflections, subsurface scattering).
|
||||
|
||||
<p align="left">
|
||||
<img src="assets/images/pipeline.png">
|
||||
</p>
|
||||
|
||||
### Performance
|
||||
|
||||
We have evaluated Hunyuan3D 2.1 with other open-source as well as close-source 3d-generation methods.
|
||||
The numerical results indicate that Hunyuan3D 2.1 surpasses all baselines in the quality of generated textured 3D assets
|
||||
and the condition following ability.
|
||||
|
||||
| Model | ULIP-T(⬆) | ULIP-I(⬆) | Uni3D-T(⬆) | Uni3D-I(⬆) |
|
||||
|-------------------------|-----------|-------------|-------------|---------------|
|
||||
| Michelangelo | 0.0752 | 0.1152 | 0.2133 | 0.2611 |
|
||||
| Craftsman | 0.0745 | 0.1296 | 0.2375 | 0.2987 |
|
||||
| TripoSG | 0.0767 | 0.1225 | 0.2506 | 0.3129 |
|
||||
| Step1X-3D | 0.0735 | 0.1183 | 0.2554 | 0.3195 |
|
||||
| Trellis | 0.0769 | 0.1267 | 0.2496 | 0.3116 |
|
||||
| Direct3D-S2 | 0.0706 | 0.1134 | 0.2346 | 0.2930 |
|
||||
| Hunyuan3D-Shape-2.1 | **0.0774** | **0.1395** | **0.2556** | **0.3213** |
|
||||
|
||||
|
||||
| Model | CLIP-FiD(⬇) | CMMD(⬇) | CLIP-I(⬆) | LPIPS(⬇) |
|
||||
|-------------------------|-----------|-------------|-------------|---------------|
|
||||
| SyncMVD-IPA | 28.39 | 2.397 | 0.8823 | 0.1423 |
|
||||
| TexGen | 28.24 | 2.448 | 0.8818 | 0.1331 |
|
||||
| Hunyuan3D-2.0 | 26.44 | 2.318 | 0.8893 | 0.1261 |
|
||||
| Hunyuan3D-Paint-2.1 | **24.78** | **2.191** | **0.9207** | **0.1211** |
|
||||
|
||||
|
||||
|
||||
## 🎁 Models Zoo
|
||||
|
||||
It takes 10 GB VRAM for shape generation, 21GB for texture generation and 29GB for shape and texture generation in total.
|
||||
|
||||
|
||||
| Model | Description | Date | Size | Huggingface |
|
||||
|----------------------------|-----------------------------|------------|------|-------------------------------------------------------------------------------------------|
|
||||
| Hunyuan3D-Shape-v2-1 | Image to Shape Model | 2025-06-14 | 3.3B | [Download](https://huggingface.co/tencent/Hunyuan3D-2.1/tree/main/hunyuan3d-dit-v2-1) |
|
||||
| Hunyuan3D-Paint-v2-1 | Texture Generation Model | 2025-06-14 | 2B | [Download](https://huggingface.co/tencent/Hunyuan3D-2.1/tree/main/hunyuan3d-paintpbr-v2-1) |
|
||||
|
||||
|
||||
## 🤗 Get Started with Hunyuan3D 2.1
|
||||
|
||||
Hunyuan3D 2.1 supports Macos, Windows, Linux. You may follow the next steps to use Hunyuan3D 2.1 via:
|
||||
|
||||
### Install Requirements
|
||||
We test our model with Python 3.10 and PyTorch 2.5.1+cu124.
|
||||
```bash
|
||||
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
|
||||
pip install -r requirements.txt
|
||||
|
||||
cd hy3dpaint/custom_rasterizer
|
||||
pip install -e .
|
||||
cd ../..
|
||||
cd hy3dpaint/DifferentiableRenderer
|
||||
bash compile_mesh_painter.sh
|
||||
cd ../..
|
||||
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P hy3dpaint/ckpt
|
||||
```
|
||||
|
||||
### Code Usage
|
||||
|
||||
We designed a diffusers-like API to use our shape generation model - Hunyuan3D-Shape and texture synthesis model -
|
||||
Hunyuan3D-Paint.
|
||||
|
||||
```python
|
||||
import sys
|
||||
sys.path.insert(0, './hy3dshape')
|
||||
sys.path.insert(0, './hy3dpaint')
|
||||
from textureGenPipeline import Hunyuan3DPaintPipeline
|
||||
from textureGenPipeline import Hunyuan3DPaintPipeline, Hunyuan3DPaintConfig
|
||||
from hy3dshape.pipelines import Hunyuan3DDiTFlowMatchingPipeline
|
||||
|
||||
# let's generate a mesh first
|
||||
shape_pipeline = Hunyuan3DDiTFlowMatchingPipeline.from_pretrained('tencent/Hunyuan3D-2.1')
|
||||
mesh_untextured = shape_pipeline(image='assets/demo.png')[0]
|
||||
|
||||
paint_pipeline = Hunyuan3DPaintPipeline(Hunyuan3DPaintConfig(max_num_view=6, resolution=512))
|
||||
mesh_textured = paint_pipeline(mesh_path, image_path='assets/demo.png')
|
||||
```
|
||||
|
||||
|
||||
### Gradio App
|
||||
|
||||
You could also host a [Gradio](https://www.gradio.app/) App in your own computer via:
|
||||
## Quick start
|
||||
|
||||
Python **3.11** (the ML stack pins old numpy; 3.12+ won't build it).
|
||||
|
||||
```bash
|
||||
python3 gradio_app.py \
|
||||
--model_path tencent/Hunyuan3D-2.1 \
|
||||
--subfolder hunyuan3d-dit-v2-1 \
|
||||
--texgen_model_path tencent/Hunyuan3D-2.1 \
|
||||
--low_vram_mode
|
||||
uv venv --python 3.11 .venv && source .venv/bin/activate # or: python3.11 -m venv .venv
|
||||
uv pip install mlx mlx-arsenal torch torchvision transformers timm \
|
||||
huggingface_hub safetensors numpy scipy scikit-image PyMCubes \
|
||||
trimesh pygltflib xatlas opencv-python pillow pymeshlab \
|
||||
einops omegaconf pyyaml tqdm torchdiffeq diffusers accelerate fast_simplification
|
||||
|
||||
export HF_HUB_DISABLE_XET=1
|
||||
python generate_e2e.py your_image.png --output out/mymesh
|
||||
```
|
||||
|
||||
Weights (~13 GB, fp16 MLX, **public — no HF login**) auto-download on first run from `dgrauet/hunyuan3d-2.1-mlx`. Override with `HUNYUAN3D_MLX_WEIGHTS_DIR`. Best results: feed a **background-removed cutout**.
|
||||
|
||||
## 🔗 BibTeX
|
||||
## Quality vs speed (generate_e2e.py flags)
|
||||
|
||||
If you found this repository helpful, please cite our reports:
|
||||
| | fast draft | **default (studio)** | max |
|
||||
|---|---|---|---|
|
||||
| `--texture-size` | 2048 | **4096** | 4096 |
|
||||
| `--remesh-faces` | 40000 | **120000** | 200000 |
|
||||
| `--octree-resolution` | 256 | **384** | 384 |
|
||||
| `--max-num-view` | 6 | 6 | 8–12 |
|
||||
| time (M3 Ultra) | ~260 s | ~380 s | longer |
|
||||
| output | 7.6 MB / 40k faces | 21.5 MB / 120k faces | larger |
|
||||
|
||||
```bibtex
|
||||
@misc{hunyuan3d2025hunyuan3d,
|
||||
title={Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Material},
|
||||
author={Tencent Hunyuan3D Team},
|
||||
year={2025},
|
||||
eprint={2506.15442},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
`--no-texture` = Stage 1 only (geometry, ~2.5 min, ~10 GB RAM).
|
||||
|
||||
@misc{hunyuan3d22025tencent,
|
||||
title={Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation},
|
||||
author={Tencent Hunyuan3D Team},
|
||||
year={2025},
|
||||
eprint={2501.12202},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
## Hardware (full table in [HARDWARE.md](HARDWARE.md))
|
||||
|
||||
@misc{yang2024hunyuan3d,
|
||||
title={Hunyuan3D 1.0: A Unified Framework for Text-to-3D and Image-to-3D Generation},
|
||||
author={Tencent Hunyuan3D Team},
|
||||
year={2024},
|
||||
eprint={2411.02293},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
Peak **~20 GB** fp16 for the full textured pipeline (measured). **Min 16 GB** (INT8/INT4) or **24 GB** (fp16, tight) · **32 GB recommended**. Any M1→M5 — fp16-MLX path, no bf16 gotcha.
|
||||
|
||||
## Acknowledgements
|
||||
Measured full textured gen: **M3 Ultra ~380 s**, **M1 Ultra ~750 s** (4096 / 120k).
|
||||
|
||||
We would like to thank the contributors to
|
||||
the [TripoSG](https://github.com/VAST-AI-Research/TripoSG), [Trellis](https://github.com/microsoft/TRELLIS), [DINOv2](https://github.com/facebookresearch/dinov2), [Stable Diffusion](https://github.com/Stability-AI/stablediffusion), [FLUX](https://github.com/black-forest-labs/flux), [diffusers](https://github.com/huggingface/diffusers), [HuggingFace](https://huggingface.co), [CraftsMan3D](https://github.com/wyysf-98/CraftsMan3D), [Michelangelo](https://github.com/NeuralCarver/Michelangelo/tree/main), [Hunyuan-DiT](https://github.com/Tencent-Hunyuan/HunyuanDiT), and [HunyuanVideo](https://github.com/Tencent-Hunyuan/HunyuanVideo) repositories, for their open research and
|
||||
exploration.
|
||||
## ⚠️ License
|
||||
|
||||
## Star History
|
||||
Inherits the **Tencent Hunyuan Non-Commercial License** (see `LICENSE` / `Notice.txt`). **Non-commercial use only** — share and use accordingly.
|
||||
|
||||
<a href="https://star-history.com/#Tencent-Hunyuan/Hunyuan3D-2.1&Date">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=Tencent-Hunyuan/Hunyuan3D-2.1&type=Date&theme=dark" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=Tencent-Hunyuan/Hunyuan3D-2.1&type=Date" />
|
||||
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=Tencent-Hunyuan/Hunyuan3D-2.1&type=Date" />
|
||||
</picture>
|
||||
</a>
|
||||
## Credits
|
||||
|
||||
Tencent Hunyuan3D team (models) · [@dgrauet](https://github.com/dgrauet) (MLX port) · MODELBEAST (this build).
|
||||
Original upstream README preserved as [`README_UPSTREAM.md`](README_UPSTREAM.md).
|
||||
|
||||
282
README_UPSTREAM.md
Normal file
282
README_UPSTREAM.md
Normal file
@ -0,0 +1,282 @@
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/images/teaser.jpg">
|
||||
</p>
|
||||
|
||||
|
||||
<div align="center">
|
||||
<a href=https://3d.hunyuan.tencent.com target="_blank"><img src=https://img.shields.io/badge/Official%20Site-333399.svg?logo=homepage height=22px></a>
|
||||
<a href=https://huggingface.co/spaces/tencent/Hunyuan3D-2.1 target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20Demo-276cb4.svg height=22px></a>
|
||||
<a href=https://huggingface.co/tencent/Hunyuan3D-2.1 target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20Models-d96902.svg height=22px></a>
|
||||
<a href=https://3d-models.hunyuan.tencent.com/ target="_blank"><img src= https://img.shields.io/badge/Page-bb8a2e.svg?logo=github height=22px></a>
|
||||
<a href=https://discord.gg/dNBrdrGGMa target="_blank"><img src= https://img.shields.io/badge/Discord-white.svg?logo=discord height=22px></a>
|
||||
<a href=https://arxiv.org/pdf/2506.15442 target="_blank"><img src=https://img.shields.io/badge/Report-b5212f.svg?logo=arxiv height=22px></a>
|
||||
<a href=https://x.com/TencentHunyuan target="_blank"><img src=https://img.shields.io/badge/Hunyuan-black.svg?logo=x height=22px></a>
|
||||
<a href="#community-resources" target="_blank"><img src=https://img.shields.io/badge/Community-lavender.svg?logo=homeassistantcommunitystore height=22px></a>
|
||||
</div>
|
||||
|
||||
[//]: # ( <a href=# target="_blank"><img src=https://img.shields.io/badge/Report-b5212f.svg?logo=arxiv height=22px></a>)
|
||||
|
||||
[//]: # ( <a href=# target="_blank"><img src= https://img.shields.io/badge/Colab-8f2628.svg?logo=googlecolab height=22px></a>)
|
||||
|
||||
[//]: # ( <a href="#"><img alt="PyPI - Downloads" src="https://img.shields.io/pypi/v/mulankit?logo=pypi" height=22px></a>)
|
||||
<br>
|
||||
|
||||
## Apple MLX Port (Apple Silicon)
|
||||
|
||||
This fork adds native **MLX inference** for the full Hunyuan3D-2.1 pipeline on Apple Silicon Macs — both shape generation (image → mesh) and PBR texture synthesis (mesh + reference image → textured GLB).
|
||||
|
||||
### Weights
|
||||
|
||||
Pre-converted MLX weights are hosted on HuggingFace and auto-downloaded on first use:
|
||||
|
||||
> [`dgrauet/hunyuan3d-2.1-mlx`](https://huggingface.co/dgrauet/hunyuan3d-2.1-mlx)
|
||||
|
||||
If you'd rather convert from the original PyTorch checkpoints yourself:
|
||||
|
||||
```bash
|
||||
pip install mlx-forge
|
||||
mlx-forge convert hunyuan3d-2.1 --output ./models/hunyuan3d-2.1-mlx
|
||||
# or quantized for 16GB Macs:
|
||||
mlx-forge convert hunyuan3d-2.1 --quantize --bits 8 --output ./models/hunyuan3d-2.1-mlx
|
||||
```
|
||||
|
||||
Override the default HF lookup by setting `HUNYUAN3D_MLX_WEIGHTS_DIR=/path/to/local/models`.
|
||||
|
||||
### Install
|
||||
|
||||
```bash
|
||||
pip install mlx mlx-arsenal safetensors Pillow trimesh scikit-image PyMCubes scipy
|
||||
pip install huggingface_hub xatlas opencv-python # for Stage 2
|
||||
```
|
||||
|
||||
### Stage 1 — Shape Generation (image → mesh)
|
||||
|
||||
```python
|
||||
from hy3dshape.hy3dshape.pipeline_mlx import ShapePipeline
|
||||
|
||||
pipe = ShapePipeline.from_pretrained("dgrauet/hunyuan3d-2.1-mlx")
|
||||
mesh = pipe("your_image.png", num_inference_steps=50, guidance_scale=7.5, octree_resolution=256)
|
||||
mesh.export("output.glb")
|
||||
```
|
||||
|
||||
### Stage 2 — PBR Texture Synthesis (mesh + reference → textured GLB)
|
||||
|
||||
```python
|
||||
from textureGenPipeline_mlx import Hunyuan3DPaintConfigMLX, Hunyuan3DPaintPipelineMLX
|
||||
|
||||
cfg = Hunyuan3DPaintConfigMLX(max_num_view=6, resolution=512)
|
||||
pipe = Hunyuan3DPaintPipelineMLX(cfg)
|
||||
pipe(
|
||||
mesh_path="mesh.glb",
|
||||
image_path="reference.png",
|
||||
output_mesh_path="textured.obj",
|
||||
save_glb=True,
|
||||
)
|
||||
```
|
||||
|
||||
Output: `textured.glb` with 2048² PBR albedo + metallic-roughness textures baked onto the mesh UVs (~9 min for 6 views at 512px on an M2 Pro).
|
||||
|
||||
Defaults match the PyTorch reference config exactly, with one documented exception: `texture_size=2048` (PT uses 4096). The MLX Metal rasterizer has no tiling and would exceed the GPU command-buffer budget at 4096² on laptop GPUs (`kIOGPUCommandBufferCallbackErrorImpactingInteractivity`). Every other knob — `max_num_view`, `render_size`, `num_inference_steps`, `guidance_scale`, bake mode, inpaint method — is PT-identical.
|
||||
|
||||
### End-to-end (image → textured GLB)
|
||||
|
||||
`tests/test_stage1_to_stage2.py` chains both stages on a single reference image. Stage 1's dense marching-cubes mesh is automatically remeshed to ~40k faces before Stage 2's bake (the Metal rasterizer can't handle Stage 1's raw ~500k faces at `texture_size=2048`).
|
||||
|
||||
### Memory Requirements
|
||||
|
||||
| Precision | DiT Size | Peak Memory | Recommended Mac |
|
||||
|-----------|----------|-------------|-----------------|
|
||||
| FP16 | 5.7 GB | ~10 GB | 32 GB+ |
|
||||
| INT8 | 3.0 GB | ~6 GB | 16 GB+ |
|
||||
| INT4 | 1.6 GB | ~4 GB | 16 GB |
|
||||
|
||||
Stage 2 (paint) adds ~6 GB peak for the UNet + VAE + DINOv2 at fp32.
|
||||
|
||||
### Scope
|
||||
|
||||
- ✅ **Stage 1** (shape generation): fully ported to MLX, validated numerically against PyTorch (1e-5)
|
||||
- ✅ **Stage 2** (PBR texture synthesis): fully ported to MLX, full UNet match within 1.17e-5 vs PyTorch
|
||||
- Metal rasterizer, UNet 2.5D with dual-stream reference attention, VAE, DINOv2, v_prediction scheduler
|
||||
- **Cosine-weighted bake blend** (PT-parity `fast_bake_texture`) smooths across face seams
|
||||
- **Mesh-aware vertex-color propagation + face barycentric raster**, `cv2.INPAINT_NS` (PT-parity), plus an EDT nearest-fill post-pass that pads UV gutters so 3D viewers doing bilinear sampling across island boundaries never pull in the atlas background
|
||||
- **RealESRGAN x4 super-resolution** in MLX (512² → 2048² per view before bake)
|
||||
- **GLB export**: PBRMaterial with sRGB `baseColorTexture`, `metallicRoughnessTexture` (glTF channel order), `doubleSided=true`
|
||||
- All pipeline defaults now match the PyTorch reference exactly (see porting principle in `docs/forward_pass.md`)
|
||||
|
||||
### Texture quality notes
|
||||
|
||||
- Reference image must match the mesh content (e.g. mermaid image on mermaid mesh). Cross-pairing produces fragmented atlases since the diffusion's albedo views can't coherently project onto an unrelated 3D layout.
|
||||
- Tested on `assets/case_1/mesh.glb` (fox) and `assets/case_2/mesh.glb` (mermaid) — all 6/6 views clean with PT-aligned defaults.
|
||||
|
||||
---
|
||||
|
||||
## 🔥 News
|
||||
|
||||
- Jul 26, 2025: 🤗 We release the first open-source, simulation-capable, immersive 3D world generation model, [HunyuanWorld-1.0](https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0)!
|
||||
- Jun 19, 2025: 👋 We present the [technical report](https://arxiv.org/pdf/2506.15442) of Hunyuan3D-2.1, please check out the details and spark some discussion!
|
||||
- Jun 13, 2025: 🤗 We release the first production-ready 3D asset generation model, Hunyuan3D-2.1!
|
||||
|
||||
> Join our **[Wechat](#)** and **[Discord](https://discord.gg/dNBrdrGGMa)** group to discuss and find help from us.
|
||||
|
||||
| Wechat Group | Xiaohongshu | X | Discord |
|
||||
|--------------------------------------------------|-------------------------------------------------------|---------------------------------------------|---------------------------------------------------|
|
||||
| <img src="assets/qrcode/wechat.png" height=140> | <img src="assets/qrcode/xiaohongshu.png" height=140> | <img src="assets/qrcode/x.png" height=140> | <img src="assets/qrcode/discord.png" height=140> |
|
||||
|
||||
## 🤗 Community Contribution Leaderboard
|
||||
1. By [@visualbruno](https://github.com/visualbruno)
|
||||
- ComfyUI-Hunyuan3d-2-1: https://github.com/visualbruno/ComfyUI-Hunyuan3d-2-1
|
||||
2. By [@VR-Jobs](https://github.com/VR-Jobs)
|
||||
- Hunyuan3d-2-1 Unity Support: https://github.com/VR-Jobs/Hunyuan3D-2.1-Unity-XR-PC-Phone
|
||||
|
||||
## ☯️ **Hunyuan3D 2.1**
|
||||
|
||||
### Architecture
|
||||
|
||||
Tencent Hunyuan3D-2.1 is a scalable 3D asset creation system that advances state-of-the-art 3D generation through two pivotal innovations: Fully Open-Source Framework and Physically-Based Rendering (PBR) Texture Synthesis. For the first time, the system releases full model weights and training code, enabling community developers to directly fine-tune and extend the model for diverse downstream applications. This transparency accelerates academic research and industrial deployment. Moreover, replacing the prior RGB-based texture model, the upgraded PBR pipeline leverages physics-grounded material simulation to generate textures with photorealistic light interaction (e.g., metallic reflections, subsurface scattering).
|
||||
|
||||
<p align="left">
|
||||
<img src="assets/images/pipeline.png">
|
||||
</p>
|
||||
|
||||
### Performance
|
||||
|
||||
We have evaluated Hunyuan3D 2.1 with other open-source as well as close-source 3d-generation methods.
|
||||
The numerical results indicate that Hunyuan3D 2.1 surpasses all baselines in the quality of generated textured 3D assets
|
||||
and the condition following ability.
|
||||
|
||||
| Model | ULIP-T(⬆) | ULIP-I(⬆) | Uni3D-T(⬆) | Uni3D-I(⬆) |
|
||||
|-------------------------|-----------|-------------|-------------|---------------|
|
||||
| Michelangelo | 0.0752 | 0.1152 | 0.2133 | 0.2611 |
|
||||
| Craftsman | 0.0745 | 0.1296 | 0.2375 | 0.2987 |
|
||||
| TripoSG | 0.0767 | 0.1225 | 0.2506 | 0.3129 |
|
||||
| Step1X-3D | 0.0735 | 0.1183 | 0.2554 | 0.3195 |
|
||||
| Trellis | 0.0769 | 0.1267 | 0.2496 | 0.3116 |
|
||||
| Direct3D-S2 | 0.0706 | 0.1134 | 0.2346 | 0.2930 |
|
||||
| Hunyuan3D-Shape-2.1 | **0.0774** | **0.1395** | **0.2556** | **0.3213** |
|
||||
|
||||
|
||||
| Model | CLIP-FiD(⬇) | CMMD(⬇) | CLIP-I(⬆) | LPIPS(⬇) |
|
||||
|-------------------------|-----------|-------------|-------------|---------------|
|
||||
| SyncMVD-IPA | 28.39 | 2.397 | 0.8823 | 0.1423 |
|
||||
| TexGen | 28.24 | 2.448 | 0.8818 | 0.1331 |
|
||||
| Hunyuan3D-2.0 | 26.44 | 2.318 | 0.8893 | 0.1261 |
|
||||
| Hunyuan3D-Paint-2.1 | **24.78** | **2.191** | **0.9207** | **0.1211** |
|
||||
|
||||
|
||||
|
||||
## 🎁 Models Zoo
|
||||
|
||||
It takes 10 GB VRAM for shape generation, 21GB for texture generation and 29GB for shape and texture generation in total.
|
||||
|
||||
|
||||
| Model | Description | Date | Size | Huggingface |
|
||||
|----------------------------|-----------------------------|------------|------|-------------------------------------------------------------------------------------------|
|
||||
| Hunyuan3D-Shape-v2-1 | Image to Shape Model | 2025-06-14 | 3.3B | [Download](https://huggingface.co/tencent/Hunyuan3D-2.1/tree/main/hunyuan3d-dit-v2-1) |
|
||||
| Hunyuan3D-Paint-v2-1 | Texture Generation Model | 2025-06-14 | 2B | [Download](https://huggingface.co/tencent/Hunyuan3D-2.1/tree/main/hunyuan3d-paintpbr-v2-1) |
|
||||
|
||||
|
||||
## 🤗 Get Started with Hunyuan3D 2.1
|
||||
|
||||
Hunyuan3D 2.1 supports Macos, Windows, Linux. You may follow the next steps to use Hunyuan3D 2.1 via:
|
||||
|
||||
### Install Requirements
|
||||
We test our model with Python 3.10 and PyTorch 2.5.1+cu124.
|
||||
```bash
|
||||
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
|
||||
pip install -r requirements.txt
|
||||
|
||||
cd hy3dpaint/custom_rasterizer
|
||||
pip install -e .
|
||||
cd ../..
|
||||
cd hy3dpaint/DifferentiableRenderer
|
||||
bash compile_mesh_painter.sh
|
||||
cd ../..
|
||||
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P hy3dpaint/ckpt
|
||||
```
|
||||
|
||||
### Code Usage
|
||||
|
||||
We designed a diffusers-like API to use our shape generation model - Hunyuan3D-Shape and texture synthesis model -
|
||||
Hunyuan3D-Paint.
|
||||
|
||||
```python
|
||||
import sys
|
||||
sys.path.insert(0, './hy3dshape')
|
||||
sys.path.insert(0, './hy3dpaint')
|
||||
from textureGenPipeline import Hunyuan3DPaintPipeline
|
||||
from textureGenPipeline import Hunyuan3DPaintPipeline, Hunyuan3DPaintConfig
|
||||
from hy3dshape.pipelines import Hunyuan3DDiTFlowMatchingPipeline
|
||||
|
||||
# let's generate a mesh first
|
||||
shape_pipeline = Hunyuan3DDiTFlowMatchingPipeline.from_pretrained('tencent/Hunyuan3D-2.1')
|
||||
mesh_untextured = shape_pipeline(image='assets/demo.png')[0]
|
||||
|
||||
paint_pipeline = Hunyuan3DPaintPipeline(Hunyuan3DPaintConfig(max_num_view=6, resolution=512))
|
||||
mesh_textured = paint_pipeline(mesh_path, image_path='assets/demo.png')
|
||||
```
|
||||
|
||||
|
||||
### Gradio App
|
||||
|
||||
You could also host a [Gradio](https://www.gradio.app/) App in your own computer via:
|
||||
|
||||
|
||||
```bash
|
||||
python3 gradio_app.py \
|
||||
--model_path tencent/Hunyuan3D-2.1 \
|
||||
--subfolder hunyuan3d-dit-v2-1 \
|
||||
--texgen_model_path tencent/Hunyuan3D-2.1 \
|
||||
--low_vram_mode
|
||||
```
|
||||
|
||||
|
||||
## 🔗 BibTeX
|
||||
|
||||
If you found this repository helpful, please cite our reports:
|
||||
|
||||
```bibtex
|
||||
@misc{hunyuan3d2025hunyuan3d,
|
||||
title={Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Material},
|
||||
author={Tencent Hunyuan3D Team},
|
||||
year={2025},
|
||||
eprint={2506.15442},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
|
||||
@misc{hunyuan3d22025tencent,
|
||||
title={Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation},
|
||||
author={Tencent Hunyuan3D Team},
|
||||
year={2025},
|
||||
eprint={2501.12202},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
|
||||
@misc{yang2024hunyuan3d,
|
||||
title={Hunyuan3D 1.0: A Unified Framework for Text-to-3D and Image-to-3D Generation},
|
||||
author={Tencent Hunyuan3D Team},
|
||||
year={2024},
|
||||
eprint={2411.02293},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
We would like to thank the contributors to
|
||||
the [TripoSG](https://github.com/VAST-AI-Research/TripoSG), [Trellis](https://github.com/microsoft/TRELLIS), [DINOv2](https://github.com/facebookresearch/dinov2), [Stable Diffusion](https://github.com/Stability-AI/stablediffusion), [FLUX](https://github.com/black-forest-labs/flux), [diffusers](https://github.com/huggingface/diffusers), [HuggingFace](https://huggingface.co), [CraftsMan3D](https://github.com/wyysf-98/CraftsMan3D), [Michelangelo](https://github.com/NeuralCarver/Michelangelo/tree/main), [Hunyuan-DiT](https://github.com/Tencent-Hunyuan/HunyuanDiT), and [HunyuanVideo](https://github.com/Tencent-Hunyuan/HunyuanVideo) repositories, for their open research and
|
||||
exploration.
|
||||
|
||||
## Star History
|
||||
|
||||
<a href="https://star-history.com/#Tencent-Hunyuan/Hunyuan3D-2.1&Date">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=Tencent-Hunyuan/Hunyuan3D-2.1&type=Date&theme=dark" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=Tencent-Hunyuan/Hunyuan3D-2.1&type=Date" />
|
||||
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=Tencent-Hunyuan/Hunyuan3D-2.1&type=Date" />
|
||||
</picture>
|
||||
</a>
|
||||
Loading…
Reference in New Issue
Block a user