bf8b1a35d5
THE DECIMATION FLOOR WAS MISDIAGNOSED. I attributed it to ~180k boundary edges. It is non-manifold edges. Measured on the shipped 500k mesh: boundary edges 32,370 NON-MANIFOLD 81,112 <- the actual blocker, 2.5x more Quadric decimation cannot collapse an edge shared by more than two faces. Upstream's own fix (fill_holes, via CUDA-only cumesh) targets boundaries and caps at max_hole_perimeter=3e-2, so it was never going to help: trimesh's equivalent moved boundaries 32,370 -> 30,990 and the floor only 214k -> 210k. That falsified it. --manifold: voxelise -> fill -> marching cubes. Removes BOTH classes at once and so closes three of the four items in one change: as shipped 499,984 faces bnd 32,370 nonmani 81,112 watertight=F IoU 0.969 remeshed 1,178,142 faces bnd 0 nonmani 0 watertight=T IoU 0.949 -> 20k 19,998 faces bnd 0 winding consistent IoU 0.956 25x smaller, fully manifold, consistent winding, for 1.3% silhouette IoU. Lossy by design - it gives up the dual grid's open-surface representation - so it is opt-in. UV BAKE is unblocked by the same change: its cost is driven by face count, not by remesh. 5.0s at 20k faces against >20min at 214k. No longer offline-only when paired with manifold. THE SCALING TRAP, worth knowing: marching_cubes returns vertices in VOXEL INDEX space. Translating without apply_scale(pitch) leaves the mesh ~292x too large. It still exports and renders as a plausible object; it silhouettes at IoU 0.08. That is how it was caught. MoGe-2 CAMERA is now wired and is the default, matching upstream; --fixed-fov keeps the old constant. It runs once per image in torch/MPS, ~0.4s after load. Reporting this one straight: it did NOT improve the samples. On 1_img, fixed 49.1 deg scored 0.893 and MoGe's 29.7 deg scored 0.883. Two caveats keep it as the default anyway - the silhouette metric projects with the SAME FOV used to generate, so a wrong-but-consistent camera can still score well and the metric cannot fully arbitrate camera correctness; and the bundled samples are synthetic renders, not the photographs MoGe reads. Real photos are the intended input here, and upstream estimates too. But the constant is one flag away and the measurement is on record rather than assumed. Operator gains manifold, divisions, fixed_fov. README and PROFILE.md corrected where they repeated the boundary-edge claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
71 lines
2.5 KiB
Python
71 lines
2.5 KiB
Python
"""Per-image camera estimation with MoGe-2.
|
|
|
|
Everything downstream assumes the object exactly fills the frame at a known FOV:
|
|
`distance_from_fov` places the camera so a unit mesh spans the image, and the proj
|
|
grid back-projects through that camera. A fixed default FOV therefore does not just
|
|
change framing — it reconstructs the subject at the WRONG DEPTH SCALE, and the failure
|
|
is quiet: you get a clean, plausible mesh that is subtly the wrong shape.
|
|
|
|
Upstream estimates FOV per image with MoGe-2 and only falls back to a constant when
|
|
`--fov` is passed explicitly. This does the same. MoGe runs once per image in torch on
|
|
MPS, like DINOv3 and NAF — outside any denoising loop, so parity beats a port.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import math
|
|
|
|
import numpy as np
|
|
|
|
MODEL_NAME = "Ruicheng/moge-2-vitl"
|
|
_model = None
|
|
|
|
|
|
def load_moge(device: str = "mps"):
|
|
"""Load MoGe-2 once per process."""
|
|
global _model
|
|
if _model is None:
|
|
import torch
|
|
from moge.model.v2 import MoGeModel
|
|
|
|
_model = MoGeModel.from_pretrained(MODEL_NAME).eval().to(device)
|
|
_model.requires_grad_(False)
|
|
elif str(next(_model.parameters()).device).split(":")[0] != device:
|
|
_model = _model.to(device)
|
|
return _model
|
|
|
|
|
|
def estimate_fov(image_path: str, device: str = "mps") -> float:
|
|
"""Horizontal FOV in radians, from MoGe-2's predicted intrinsics.
|
|
|
|
`intrinsics[0,0]` is fx NORMALISED by image width, so it must be multiplied back
|
|
up before the arctan — the normalisation is easy to miss and yields a plausible
|
|
but wrong angle if skipped.
|
|
"""
|
|
import torch
|
|
from PIL import Image
|
|
|
|
img = Image.open(image_path).convert("RGB")
|
|
width, _ = img.size
|
|
arr = np.asarray(img, dtype=np.float32) / 255.0
|
|
tensor = torch.from_numpy(arr).permute(2, 0, 1).to(device)
|
|
|
|
model = load_moge(device)
|
|
with torch.no_grad():
|
|
out = model.infer(tensor)
|
|
fx_norm = float(np.asarray(out["intrinsics"].squeeze().cpu())[0, 0])
|
|
return 2.0 * math.atan(width / (2.0 * fx_norm * width))
|
|
|
|
|
|
def camera_for(image_path: str, fov: float | None = None, mesh_scale: float = 1.0,
|
|
image_resolution: int = 512, device: str = "mps") -> dict:
|
|
"""(camera_angle_x, distance) for an image. Estimates FOV when `fov` is None."""
|
|
from .proj import distance_from_fov
|
|
|
|
estimated = fov is None
|
|
if estimated:
|
|
fov = estimate_fov(image_path, device)
|
|
return {"camera_angle_x": float(fov),
|
|
"distance": distance_from_fov(fov, mesh_scale, image_resolution),
|
|
"estimated": estimated}
|