165de26db5
image_to_mesh() now runs the real cascade, not the single-stage shortcut: structure 3048 voxels @32^3 (64^3 occupancy, MAX-POOLED DOWN) LR SLAT 3048 x 32 shape_512 extractor refine 13147 coords @64^3 four decoder stages -> coords -> quantise HR SLAT 13147 x 32 shape_1024 extractor mesh 3988052 verts, 7996876 faces @1024^3 TOTAL 258.4s, peak 27.9GB with every model resident silhouette IoU 0.969 Three things the cascade needed: 1. occupied_coords_at() - ss_dec always decodes 64^3 but the cascade STARTS at 32^3. Upstream max-pools the boolean grid down by the ratio (a voxel survives if ANY of its eight children was occupied). I had been feeding the raw 64^3 set to the HR flow. 2. decoder.upsample() - pushes the LR latent four stages in and returns COORDS, not features. The predicted subdivisions grow the occupied set; those coords quantise onto the HR flow's grid. Stops BEFORE stage `upsample_times`, as upstream does; one stage further doubles the resolution and misplaces every voxel. 3. grid_resolution override on ProjConditioner - upstream backs the HR grid off in 128-unit steps while the token count exceeds max_num_tokens, so a dense object degrades instead of exploding. refine_coords() implements that loop. I WAS WRONG ABOUT THE HALO. The previous commit blamed the single-stage shortcut for a 0.639 silhouette IoU and predicted the cascade would fix it. The cascade measured 0.640 - no change. The real fault was in my VERIFICATION, not the pipeline: o_voxel returns vertices in the voxel-grid frame, while ProjGrid rotates its lattice by _BLENDER_ROT before projecting. Rotating the mesh the same way scores 0.969 on the same geometry the earlier commit had already produced. Added mesh.to_camera_frame() so the trap is named where it bites; the earlier mesh was correct all along. The cascade is still the right thing - it is the shipped path, and staged loading halves peak memory (12.8GB vs 22.6GB) when models are released between stages. Also adds models.load_all(), so a server builds all five models plus both conditioners ONCE. Warmup is ~71s against ~17s of compute, so an operator must never fork per job. Holding everything resident costs 27.9GB peak - nothing on a 256GB box. scripts/image_to_mesh.py exits non-zero if IoU < 0.85: a run that completes with a bad reconstruction has failed even though nothing raised. 27/27 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
93 lines
3.4 KiB
Python
93 lines
3.4 KiB
Python
"""Image -> GLB through the full Pixal3D cascade, with a silhouette check.
|
|
|
|
Usage: python scripts/image_to_mesh.py IMAGE [-o OUT.glb] [--fov RAD] [--seed N]
|
|
|
|
The silhouette IoU is the acceptance test: re-project the mesh through the same camera
|
|
and compare against the input matte. Pixal3D's entire claim is pixel alignment, so a
|
|
run that completes with a poor IoU has failed even though nothing raised.
|
|
"""
|
|
|
|
import argparse
|
|
import sys
|
|
import time
|
|
from pathlib import Path
|
|
|
|
import mlx.core as mx
|
|
import numpy as np
|
|
import trimesh
|
|
|
|
REPO = Path(__file__).resolve().parents[1]
|
|
sys.path.insert(0, str(REPO))
|
|
|
|
from pixal3d_mlx.mesh import to_camera_frame # noqa: E402
|
|
from pixal3d_mlx.models import load_all, normalization # noqa: E402
|
|
from pixal3d_mlx.pipeline import DEFAULT_FOV, image_to_mesh # noqa: E402
|
|
|
|
DEFAULT_IMAGE = REPO / "upstream" / "Pixal3D" / "assets" / "images" / "0_img.png"
|
|
|
|
|
|
def silhouette_iou(vertices, image_path, fov, res=512):
|
|
"""Re-project the mesh through the generating camera; IoU against the input matte.
|
|
|
|
Vertices MUST be rotated into the camera frame first — o_voxel returns them in the
|
|
voxel-grid frame while ProjGrid rotates its lattice before projecting.
|
|
"""
|
|
from PIL import Image
|
|
from scipy.ndimage import binary_dilation
|
|
|
|
from pixal3d_mlx.cond import preprocess_image
|
|
from pixal3d_mlx.proj import _FRONT_VIEW, distance_from_fov, project_points
|
|
|
|
img = Image.open(image_path)
|
|
if img.mode != "RGBA":
|
|
return None
|
|
matte = np.asarray(preprocess_image(img).convert("L").resize((res, res))) > 8
|
|
|
|
tm = _FRONT_VIEW.copy()
|
|
tm[1, 3] = -distance_from_fov(fov, 1.0, res)
|
|
pts = to_camera_frame(vertices).astype(np.float32)[None]
|
|
px, _, _ = project_points(mx.array(pts), mx.array(tm[None]), fov, res)
|
|
px = np.asarray(px)[0].astype(int)
|
|
|
|
keep = (px[:, 0] >= 0) & (px[:, 0] < res) & (px[:, 1] >= 0) & (px[:, 1] < res)
|
|
proj = np.zeros((res, res), bool)
|
|
proj[px[keep, 1], px[keep, 0]] = True
|
|
proj = binary_dilation(proj, np.ones((3, 3), bool))
|
|
return (proj & matte).sum() / (proj | matte).sum(), proj, matte
|
|
|
|
|
|
def main():
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("image", nargs="?", default=str(DEFAULT_IMAGE))
|
|
ap.add_argument("-o", "--output", default=str(REPO / "output.glb"))
|
|
ap.add_argument("--fov", type=float, default=DEFAULT_FOV)
|
|
ap.add_argument("--seed", type=int, default=0)
|
|
ap.add_argument("--no-check", action="store_true")
|
|
a = ap.parse_args()
|
|
|
|
t = time.time()
|
|
models = load_all()
|
|
print(f"loaded models ({time.time() - t:.1f}s, lazy — weights fault in on first use)")
|
|
|
|
v, f, info = image_to_mesh(a.image, models, camera_angle_x=a.fov, seed=a.seed,
|
|
normalization=normalization("shape"))
|
|
|
|
mesh = trimesh.Trimesh(v.cpu().numpy(), f.cpu().numpy(), process=False)
|
|
mesh.export(a.output)
|
|
print(f"\nTOTAL {info['seconds']}s peak {info['peak_gb']} GB -> {a.output}")
|
|
|
|
if not a.no_check:
|
|
got = silhouette_iou(mesh.vertices, a.image, a.fov)
|
|
if got is None:
|
|
print("(input has no alpha matte — skipping silhouette check)")
|
|
else:
|
|
iou, _, _ = got
|
|
print(f"silhouette IoU {iou:.3f}")
|
|
if iou < 0.85:
|
|
print("WARNING: low IoU — the reconstruction is not tracking the input")
|
|
sys.exit(1)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|