bf8b1a35d5
THE DECIMATION FLOOR WAS MISDIAGNOSED. I attributed it to ~180k boundary edges. It is non-manifold edges. Measured on the shipped 500k mesh: boundary edges 32,370 NON-MANIFOLD 81,112 <- the actual blocker, 2.5x more Quadric decimation cannot collapse an edge shared by more than two faces. Upstream's own fix (fill_holes, via CUDA-only cumesh) targets boundaries and caps at max_hole_perimeter=3e-2, so it was never going to help: trimesh's equivalent moved boundaries 32,370 -> 30,990 and the floor only 214k -> 210k. That falsified it. --manifold: voxelise -> fill -> marching cubes. Removes BOTH classes at once and so closes three of the four items in one change: as shipped 499,984 faces bnd 32,370 nonmani 81,112 watertight=F IoU 0.969 remeshed 1,178,142 faces bnd 0 nonmani 0 watertight=T IoU 0.949 -> 20k 19,998 faces bnd 0 winding consistent IoU 0.956 25x smaller, fully manifold, consistent winding, for 1.3% silhouette IoU. Lossy by design - it gives up the dual grid's open-surface representation - so it is opt-in. UV BAKE is unblocked by the same change: its cost is driven by face count, not by remesh. 5.0s at 20k faces against >20min at 214k. No longer offline-only when paired with manifold. THE SCALING TRAP, worth knowing: marching_cubes returns vertices in VOXEL INDEX space. Translating without apply_scale(pitch) leaves the mesh ~292x too large. It still exports and renders as a plausible object; it silhouettes at IoU 0.08. That is how it was caught. MoGe-2 CAMERA is now wired and is the default, matching upstream; --fixed-fov keeps the old constant. It runs once per image in torch/MPS, ~0.4s after load. Reporting this one straight: it did NOT improve the samples. On 1_img, fixed 49.1 deg scored 0.893 and MoGe's 29.7 deg scored 0.883. Two caveats keep it as the default anyway - the silhouette metric projects with the SAME FOV used to generate, so a wrong-but-consistent camera can still score well and the metric cannot fully arbitrate camera correctness; and the bundled samples are synthetic renders, not the photographs MoGe reads. Real photos are the intended input here, and upstream estimates too. But the constant is one flag away and the measurement is on record rather than assumed. Operator gains manifold, divisions, fixed_fov. README and PROFILE.md corrected where they repeated the boundary-edge claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5.3 KiB
Profile — and why the two planned optimisations are not worth doing
Measured on m3ultra (M3 Ultra 256GB), 0_img.png, full textured cascade, MLX 0.32.
The plan going in was to port two things down from trellis-2-mrp-mlx into the shared
sparse core: the fused Metal spconv kernel and the 15 mx.compile sites. The
profile says neither pays. Recording the numbers so nobody re-opens it on a hunch.
Stage breakdown
| stage | seconds | peak GB | backend |
|---|---|---|---|
| ss_flow + ss_dec | 18.0 | 6.7 | DiT + dense Conv3d |
| cond_512 (DINOv3) | 1.6 | 6.7 | torch/MPS |
| LR slat_flow | 12.5 | 12.4 | DiT |
| refine (partial decode) | 3.4 | 15.5 | sparse conv |
| cond_1024 + NAF | 0.9 | 15.5 | torch/MPS |
| HR slat_flow | 90.9 | 23.6 | DiT |
| shape_dec | 11.6 | 28.4 | sparse conv |
| fdg_to_mesh | 0.8 | 28.4 | o_voxel native |
| cond_tex + NAF | 1.6 | 28.4 | torch/MPS |
| tex stage | 63.0 | 37.1 | 51.3 DiT + 11.1 sparse |
| TOTAL | 204.3 | 37.1 |
Splitting the tex stage by hand (it is the only mixed one): 51.3s flow / 11.1s decoder, so sparse conv is 17.8% of it.
By backend
| backend | seconds | share |
|---|---|---|
| DiT (transformer flows) | ~173 | ~85% |
| sparse conv (decoders) | ~26 | ~13% |
| torch/MPS (DINOv3 + NAF) | 4.1 | 2% |
| o_voxel native | 0.8 | 0.4% |
The four flow models contain ZERO sparse-conv tensors — verified by counting 5-D
tensors in the checkpoints (700/700 params each, none 5-D). Only shape_dec (40),
tex_dec (40) and ss_dec (20, dense Conv3d) use them. So the two candidate ports
land on completely disjoint parts of the pipeline, and one of those parts is 13%.
Why the Metal spconv kernel is not the win
It can only touch the ~13% in the decoders. Its own docstring says the prize is peak memory, not time — it was written because 128GB gated the M1 Ultra. Even a free 2x on all sparse conv would return ~13s of 204s (6%).
It remains the right thing to port if peak memory ever matters — 37.1GB is fine on a 256GB Ultra and impossible on a 24GB M4 Pro (see below).
Why mx.compile is not the win either
The DiT loop is compute-bound, not dispatch-bound. Two measurements:
Scaling is super-linear in tokens — doubling tokens costs 2.4–2.8x, which is the O(n²) attention, not launch overhead:
25% tokens ( 3,286) 646 ms 0.15x
50% tokens ( 6,573) 1,532 ms 0.36x
100% tokens (13,147) 4,261 ms 1.00x
Per-block FLOPs confirm attention dominates:
self-attention 1062 GFLOP 58.8% <- O(n^2)
MLP 496 GFLOP 27.5%
qkv+out proj 248 GFLOP 13.7%
total 1806 GFLOP
at 142 ms/block -> 12.7 TFLOP/s achieved
12.7 TFLOP/s is a respectable fraction of what an M3 Ultra delivers on bf16 weights
with fp32 accumulation, and attention already routes through
mx.fast.scaled_dot_product_attention — the fast path.
Direct test on the compilable part (the norm+MLP chain, pure arrays):
eager 23.9 ms
mx.compile 23.7 ms <- 0.8%, i.e. nothing
There is also a structural obstacle: mx.compile wants pure array-in/array-out
functions, while the sparse path threads SparseTensor objects carrying Python-side
layout. Compiling the blocks would mean restructuring that, for a measured ~0%.
What WOULD move the needle
- Fewer tokens. Attention is O(n²), so token count is the dominant lever — 50%
of the tokens ran 2.8x faster.
refine_coordsalready backs the grid off when the count exceedsmax_num_tokens(49152); lowering that knob is the real speed control, and it trades resolution for time honestly. - Quantization, for memory and possibly bandwidth. Not attempted. Treat with suspicion: a published field report on ARDY found INT8 preserved embedding cosine at 0.992+ while the generated output diverged badly, so it would need output-level validation (the silhouette gate is well suited to that), not a similarity metric.
- The Metal spconv kernel — for memory only, if this ever needs to run somewhere smaller than a Studio.
Correction: the decimation floor was misdiagnosed
An earlier version of this file blamed the ~214k floor on "~180,000 boundary edges". That was wrong. Measured on the shipped 500k mesh:
boundary edges 32,370
NON-MANIFOLD 81,112 <- the actual blocker, 2.5x more
Quadric decimation cannot collapse an edge shared by more than two faces. Filling holes moved boundaries 32,370 → 30,990 and the floor only 214k → 210k, which is what falsified the boundary theory.
--manifold (voxel remesh) removes both classes: 0 boundary, 0 non-manifold,
watertight, winding consistent, and it decimates to 20k faces at IoU 0.956 against
0.969 for the 500k non-manifold original. It also drops the UV bake from >20 minutes
to 5.0 seconds, since xatlas was choking on face count.
Watch for the scaling trap: marching_cubes returns vertices in VOXEL INDEX space.
Translating without apply_scale(pitch) leaves the mesh ~292x too large — it still
exports and renders as a plausible object and silhouettes at IoU 0.08.
m4pro cannot run this
Peak is 37.1 GB. The m4pro is 24 GB, so the full cascade will not fit, and the weights alone are 24 GB before activations. Geometry-only peaks lower (~28 GB) and still does not fit. Not a tuning problem — a capacity one.