bf8b1a35d5
THE DECIMATION FLOOR WAS MISDIAGNOSED. I attributed it to ~180k boundary edges. It is non-manifold edges. Measured on the shipped 500k mesh: boundary edges 32,370 NON-MANIFOLD 81,112 <- the actual blocker, 2.5x more Quadric decimation cannot collapse an edge shared by more than two faces. Upstream's own fix (fill_holes, via CUDA-only cumesh) targets boundaries and caps at max_hole_perimeter=3e-2, so it was never going to help: trimesh's equivalent moved boundaries 32,370 -> 30,990 and the floor only 214k -> 210k. That falsified it. --manifold: voxelise -> fill -> marching cubes. Removes BOTH classes at once and so closes three of the four items in one change: as shipped 499,984 faces bnd 32,370 nonmani 81,112 watertight=F IoU 0.969 remeshed 1,178,142 faces bnd 0 nonmani 0 watertight=T IoU 0.949 -> 20k 19,998 faces bnd 0 winding consistent IoU 0.956 25x smaller, fully manifold, consistent winding, for 1.3% silhouette IoU. Lossy by design - it gives up the dual grid's open-surface representation - so it is opt-in. UV BAKE is unblocked by the same change: its cost is driven by face count, not by remesh. 5.0s at 20k faces against >20min at 214k. No longer offline-only when paired with manifold. THE SCALING TRAP, worth knowing: marching_cubes returns vertices in VOXEL INDEX space. Translating without apply_scale(pitch) leaves the mesh ~292x too large. It still exports and renders as a plausible object; it silhouettes at IoU 0.08. That is how it was caught. MoGe-2 CAMERA is now wired and is the default, matching upstream; --fixed-fov keeps the old constant. It runs once per image in torch/MPS, ~0.4s after load. Reporting this one straight: it did NOT improve the samples. On 1_img, fixed 49.1 deg scored 0.893 and MoGe's 29.7 deg scored 0.883. Two caveats keep it as the default anyway - the silhouette metric projects with the SAME FOV used to generate, so a wrong-but-consistent camera can still score well and the metric cannot fully arbitrate camera correctness; and the bundled samples are synthetic renders, not the photographs MoGe reads. Real photos are the intended input here, and upstream estimates too. But the constant is one flag away and the measurement is on record rather than assumed. Operator gains manifold, divisions, fixed_fov. README and PROFILE.md corrected where they repeated the boundary-edge claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
130 lines
5.3 KiB
Markdown
130 lines
5.3 KiB
Markdown
# Profile — and why the two planned optimisations are not worth doing
|
||
|
||
Measured on m3ultra (M3 Ultra 256GB), `0_img.png`, full textured cascade, MLX 0.32.
|
||
|
||
The plan going in was to port two things down from `trellis-2-mrp-mlx` into the shared
|
||
sparse core: the **fused Metal spconv kernel** and the **15 `mx.compile` sites**. The
|
||
profile says neither pays. Recording the numbers so nobody re-opens it on a hunch.
|
||
|
||
## Stage breakdown
|
||
|
||
| stage | seconds | peak GB | backend |
|
||
|---|---|---|---|
|
||
| ss_flow + ss_dec | 18.0 | 6.7 | DiT + dense Conv3d |
|
||
| cond_512 (DINOv3) | 1.6 | 6.7 | torch/MPS |
|
||
| LR slat_flow | 12.5 | 12.4 | DiT |
|
||
| refine (partial decode) | 3.4 | 15.5 | sparse conv |
|
||
| cond_1024 + NAF | 0.9 | 15.5 | torch/MPS |
|
||
| **HR slat_flow** | **90.9** | 23.6 | DiT |
|
||
| shape_dec | 11.6 | 28.4 | sparse conv |
|
||
| fdg_to_mesh | 0.8 | 28.4 | o_voxel native |
|
||
| cond_tex + NAF | 1.6 | 28.4 | torch/MPS |
|
||
| **tex stage** | **63.0** | **37.1** | 51.3 DiT + 11.1 sparse |
|
||
| **TOTAL** | **204.3** | **37.1** | |
|
||
|
||
Splitting the tex stage by hand (it is the only mixed one): **51.3s flow / 11.1s
|
||
decoder**, so sparse conv is 17.8% of it.
|
||
|
||
## By backend
|
||
|
||
| backend | seconds | share |
|
||
|---|---|---|
|
||
| **DiT (transformer flows)** | **~173** | **~85%** |
|
||
| sparse conv (decoders) | ~26 | ~13% |
|
||
| torch/MPS (DINOv3 + NAF) | 4.1 | 2% |
|
||
| o_voxel native | 0.8 | 0.4% |
|
||
|
||
**The four flow models contain ZERO sparse-conv tensors** — verified by counting 5-D
|
||
tensors in the checkpoints (700/700 params each, none 5-D). Only `shape_dec` (40),
|
||
`tex_dec` (40) and `ss_dec` (20, dense Conv3d) use them. So the two candidate ports
|
||
land on completely disjoint parts of the pipeline, and one of those parts is 13%.
|
||
|
||
## Why the Metal spconv kernel is not the win
|
||
|
||
It can only touch the ~13% in the decoders. Its own docstring says the prize is **peak
|
||
memory, not time** — it was written because 128GB gated the M1 Ultra. Even a free 2x
|
||
on all sparse conv would return ~13s of 204s (6%).
|
||
|
||
It remains the right thing to port **if peak memory ever matters** — 37.1GB is fine on
|
||
a 256GB Ultra and impossible on a 24GB M4 Pro (see below).
|
||
|
||
## Why `mx.compile` is not the win either
|
||
|
||
The DiT loop is **compute-bound, not dispatch-bound**. Two measurements:
|
||
|
||
**Scaling is super-linear in tokens** — doubling tokens costs 2.4–2.8x, which is the
|
||
O(n²) attention, not launch overhead:
|
||
|
||
```
|
||
25% tokens ( 3,286) 646 ms 0.15x
|
||
50% tokens ( 6,573) 1,532 ms 0.36x
|
||
100% tokens (13,147) 4,261 ms 1.00x
|
||
```
|
||
|
||
**Per-block FLOPs confirm attention dominates:**
|
||
|
||
```
|
||
self-attention 1062 GFLOP 58.8% <- O(n^2)
|
||
MLP 496 GFLOP 27.5%
|
||
qkv+out proj 248 GFLOP 13.7%
|
||
total 1806 GFLOP
|
||
at 142 ms/block -> 12.7 TFLOP/s achieved
|
||
```
|
||
|
||
12.7 TFLOP/s is a respectable fraction of what an M3 Ultra delivers on bf16 weights
|
||
with fp32 accumulation, and attention already routes through
|
||
`mx.fast.scaled_dot_product_attention` — the fast path.
|
||
|
||
**Direct test on the compilable part** (the norm+MLP chain, pure arrays):
|
||
|
||
```
|
||
eager 23.9 ms
|
||
mx.compile 23.7 ms <- 0.8%, i.e. nothing
|
||
```
|
||
|
||
There is also a structural obstacle: `mx.compile` wants pure array-in/array-out
|
||
functions, while the sparse path threads `SparseTensor` objects carrying Python-side
|
||
layout. Compiling the blocks would mean restructuring that, for a measured ~0%.
|
||
|
||
## What WOULD move the needle
|
||
|
||
1. **Fewer tokens.** Attention is O(n²), so token count is the dominant lever — 50%
|
||
of the tokens ran 2.8x faster. `refine_coords` already backs the grid off when the
|
||
count exceeds `max_num_tokens` (49152); lowering that knob is the real speed
|
||
control, and it trades resolution for time honestly.
|
||
2. **Quantization**, for memory and possibly bandwidth. Not attempted. Treat with
|
||
suspicion: a published field report on ARDY found INT8 preserved embedding cosine
|
||
at 0.992+ while the generated output diverged badly, so it would need output-level
|
||
validation (the silhouette gate is well suited to that), not a similarity metric.
|
||
3. **The Metal spconv kernel — for memory only**, if this ever needs to run somewhere
|
||
smaller than a Studio.
|
||
|
||
## Correction: the decimation floor was misdiagnosed
|
||
|
||
An earlier version of this file blamed the ~214k floor on "~180,000 boundary edges".
|
||
That was wrong. Measured on the shipped 500k mesh:
|
||
|
||
```
|
||
boundary edges 32,370
|
||
NON-MANIFOLD 81,112 <- the actual blocker, 2.5x more
|
||
```
|
||
|
||
Quadric decimation cannot collapse an edge shared by more than two faces. Filling
|
||
holes moved boundaries 32,370 → 30,990 and the floor only 214k → 210k, which is what
|
||
falsified the boundary theory.
|
||
|
||
`--manifold` (voxel remesh) removes both classes: 0 boundary, 0 non-manifold,
|
||
watertight, winding consistent, and it decimates to **20k faces at IoU 0.956** against
|
||
0.969 for the 500k non-manifold original. It also drops the **UV bake from >20 minutes
|
||
to 5.0 seconds**, since xatlas was choking on face count.
|
||
|
||
Watch for the scaling trap: `marching_cubes` returns vertices in VOXEL INDEX space.
|
||
Translating without `apply_scale(pitch)` leaves the mesh ~292x too large — it still
|
||
exports and renders as a plausible object and silhouettes at IoU 0.08.
|
||
|
||
## m4pro cannot run this
|
||
|
||
Peak is **37.1 GB**. The m4pro is 24 GB, so the full cascade will not fit, and the
|
||
weights alone are 24 GB before activations. Geometry-only peaks lower (~28 GB) and
|
||
still does not fit. Not a tuning problem — a capacity one.
|