The everything-on job failed the gate at IoU 0.660 on geometry that measures 0.956.
Cause: o_voxel's to_glb applies _BLENDER_ROT on export, (x,y,z) -> (x, z, -y), which
is EXACTLY the inverse of to_camera_frame. So the vertex baker and the UV baker were
returning meshes in different frames and the gate double-rotated the UV one.
Verified numerically: OV == _BLENDER_ROT, and OV @ _BLENDER_ROT.T == I.
to_glb now un-rotates back into the voxel-grid frame and returns a Trimesh, so every
path in mesh.py speaks one frame. After the fix the baked mesh measures IoU 0.883 --
identical to its input -- with bounds matching to 3dp.
That is the THIRD frame bug this session (mesh vertices vs ProjGrid's rotated lattice;
marching_cubes' voxel-index space; now o_voxel's export rotation). None of them throw:
each produces a plausible object that renders fine and silhouettes wrong. The gate
caught all three, which is the argument for having it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
THE DECIMATION FLOOR WAS MISDIAGNOSED. I attributed it to ~180k boundary edges. It is
non-manifold edges. Measured on the shipped 500k mesh:
boundary edges 32,370
NON-MANIFOLD 81,112 <- the actual blocker, 2.5x more
Quadric decimation cannot collapse an edge shared by more than two faces. Upstream's
own fix (fill_holes, via CUDA-only cumesh) targets boundaries and caps at
max_hole_perimeter=3e-2, so it was never going to help: trimesh's equivalent moved
boundaries 32,370 -> 30,990 and the floor only 214k -> 210k. That falsified it.
--manifold: voxelise -> fill -> marching cubes. Removes BOTH classes at once and so
closes three of the four items in one change:
as shipped 499,984 faces bnd 32,370 nonmani 81,112 watertight=F IoU 0.969
remeshed 1,178,142 faces bnd 0 nonmani 0 watertight=T IoU 0.949
-> 20k 19,998 faces bnd 0 winding consistent IoU 0.956
25x smaller, fully manifold, consistent winding, for 1.3% silhouette IoU. Lossy by
design - it gives up the dual grid's open-surface representation - so it is opt-in.
UV BAKE is unblocked by the same change: its cost is driven by face count, not by
remesh. 5.0s at 20k faces against >20min at 214k. No longer offline-only when paired
with manifold.
THE SCALING TRAP, worth knowing: marching_cubes returns vertices in VOXEL INDEX space.
Translating without apply_scale(pitch) leaves the mesh ~292x too large. It still
exports and renders as a plausible object; it silhouettes at IoU 0.08. That is how it
was caught.
MoGe-2 CAMERA is now wired and is the default, matching upstream; --fixed-fov keeps
the old constant. It runs once per image in torch/MPS, ~0.4s after load.
Reporting this one straight: it did NOT improve the samples. On 1_img, fixed 49.1 deg
scored 0.893 and MoGe's 29.7 deg scored 0.883. Two caveats keep it as the default
anyway - the silhouette metric projects with the SAME FOV used to generate, so a
wrong-but-consistent camera can still score well and the metric cannot fully arbitrate
camera correctness; and the bundled samples are synthetic renders, not the photographs
MoGe reads. Real photos are the intended input here, and upstream estimates too. But
the constant is one flag away and the measurement is on record rather than assumed.
Operator gains manifold, divisions, fixed_fov. README and PROFILE.md corrected where
they repeated the boundary-edge claim.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The plan was to port the fused Metal spconv kernel and the 15 mx.compile sites down
from trellis-2-mrp-mlx into the shared core. Measured on the full textured cascade,
both are dead ends. PROFILE.md has the numbers; the short version:
BY BACKEND (204.3s total, peak 37.1GB)
DiT transformer flows ~173s ~85%
sparse conv (decoders) ~26s ~13%
torch/MPS (DINOv3+NAF) 4.1s 2%
o_voxel native 0.8s 0.4%
The Metal kernel can only touch the 13%. Verified from the checkpoints: the four flow
models contain ZERO 5-D tensors, so they never call sparse conv at all - only shape_dec
(40), tex_dec (40) and ss_dec (20, dense) do. A free 2x on ALL sparse conv returns 6%
of runtime. The kernel is still the right port if peak memory ever matters, which is
its stated prize, but it is not a speed fix.
mx.compile is no better, because the DiT loop is compute-bound rather than
dispatch-bound - the opposite of the launch-latency problem that motivated the fused
gather-matmul in the shared core:
scaling 25% tokens -> 0.15x, 50% -> 0.36x, 100% -> 1.00x (super-linear)
per block attention 1062 GFLOP 58.8% (O(n^2)), MLP 496, proj 248
12.7 TFLOP/s achieved at 142 ms/block
direct test eager 23.9 ms vs mx.compile 23.7 ms -> 0.8%, i.e. nothing
Attention already routes through mx.fast.scaled_dot_product_attention. mx.compile also
wants pure array-in/array-out functions while the sparse path threads SparseTensor
objects with Python-side layout, so it would mean restructuring for a measured ~0%.
The real lever is TOKEN COUNT: attention is O(n^2), half the tokens ran 2.8x faster,
and refine_coords already backs the grid off past max_num_tokens. That knob trades
resolution for time honestly.
m4pro cannot run this: peak 37.1GB against 24GB of RAM, and the weights alone are 24GB
before activations. Capacity, not tuning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two bugs in the acceptance test, both found by running it for real.
1. THE GATE DID NOT APPLY TO TEXTURED RUNS. The texture path returned before the
check, so the one path most likely to be used for real assets was the only one that
could ship a blob silently. The check is now a shared gate() called from both.
2. THE METRIC WAS TESSELLATION-DEPENDENT. It projected the VERTEX LIST, so a decimated
mesh sampled its own silhouette more sparsely and scored lower for an identical
shape - holes appear inside the outline and count as misses. Measured on two real
assets:
asset verts vertex-proj surface-sampled
1_img (gate FAILED) 133,842 0.790 0.911
0_img (gate passed) 227,546 0.965 0.969
The dense mesh barely moves; the sparse one jumps 0.12. That is the metric
measuring tessellation, not accuracy - and at min_iou 0.85 it had just rejected a
good reconstruction. Overlay confirmed it: the "missing" region was speckle inside
the silhouette, not a wrong shape.
Now samples 3M points uniformly over the surface, so density is a constant of the
metric rather than a property of the mesh.
Worth stating plainly: the gate caught a real problem on its first live failure - just
not the one it reported. A quality gate that is itself unvalidated is a liability, and
this one needed the same "measure it, do not reason about it" treatment as the model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The texture stage produced correct PBR voxels, but getting them ONTO a mesh through
o_voxel's UV path is not viable on this build. Measured, on an already welded,
floater-free, decimated 214k-face mesh:
o_voxel to_glb, remesh=True killed at 20min
o_voxel to_glb, remesh=False >20min CPU, killed
bake_vertex_colors 0.2s
xatlas scales badly and 214k is the decimation floor, so it cannot be fed a smaller
mesh either. The trellis-2 lane reached the same conclusion independently and ships
--baker vertex as its fast path; this now matches.
bake_vertex_colors samples the PBR attribute volume at each vertex and writes COLOR_0.
Positions map to voxel indices by the same linear aabb relation fdg_to_mesh uses, so
nothing is resampled; lookup is a sorted-key searchsorted, and misses keep neutral
grey rather than black.
Verified on the real pipeline output:
bake vertex colours, 96.1% of vertices hit 0.2s
result 90,093 verts / 214,322 faces
57,925 unique colours, mean RGB [105 100 87], std [40 35 38]
3.9% still default grey (matches the 4% miss rate)
TOTAL 266.2s end to end, peak 32.6GB
What this costs: no metallic/roughness maps, base colour only. That is the honest
trade and it is stated in the operator description rather than buried - remesh also
now defaults OFF in to_glb, since upstream's remesh=True assumes CUDA.
Operator gains a baker param (vertex default, uv opt-in and flagged offline-only).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The texture flow is imgshape2tex - it denoises 32 PBR channels while SEEING the shape
latent, so in_channels is 64 against out_channels 32. Upstream feeds the shape latent
as concat_cond and the model does sparse_cat([x, concat_cond], dim=-1); both share
coords, so it reduces to a channel concat. Added to slat_flow and carried on the
sampler (it is fixed for the whole trajectory and must reach BOTH CFG branches).
Verified running on the real checkpoints: 3,988,052 PBR voxels x 6 channels in 65.8s
(base_color 0:3, metallic 3:4, roughness 4:5, alpha 5:6).
Two things here fail SILENTLY rather than loudly, so both are asserted in comments:
1. shape_slat arrives DENORMALISED - the shape stage un-standardises it for the
decoder - but the texture flow was trained against the standardised form. It is
re-normalised before use as concat_cond. Skipping that gives a plausible mesh with
wrong colours, not an error.
2. tex_dec has pred_subdiv=False: it cannot invent subdivisions and must be handed the
shape decoder's subs as guides, so texture voxels land on the geometry that was
actually built.
The decoder's output is mapped * 0.5 + 0.5 into [0,1], the range o_voxel expects.
BAKE ORDER. Handing o_voxel the raw ~8M-face mesh hangs - the same wall the standalone
remesh test hit (killed at 20min), and the trellis2 lane's own operator note says the
uncapped bake peaks at 75GB. So the mesh is welded, stripped of floaters and decimated
BEFORE baking; the baker samples the attribute VOLUME at mesh positions, so a
decimated mesh still gets correct colours. Measured on the way through:
welded 3,988,052 -> 3,983,672 verts
floaters 12 components -> 1 kept, 6,332 faces dropped
decimated 7,996,876 -> 214,322 faces
pre-bake 34.6s
That floater count is worth noting: 12 components, not the 52,855 the first health
pass reported. Welding first is what makes the difference.
remesh now defaults OFF in to_glb, unlike upstream. Upstream runs on CUDA; this is the
CPU/Metal build and its remesher took >20 minutes on a 214k-face mesh. It is also
handed an already-clean mesh, so there is far less for it to fix.
Operator gains texture + texture_size params; geometry-only stays the default because
it is ~3min against the textured path's extra flow and bake.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The status list still said the pipeline was unwired. It runs end to end now:
proj conditioning (all four extractors at corr 1.00000000), the NAF branch without
natten, the full 32^3 -> refine -> 64^3 cascade at silhouette IoU 0.969, cleanup,
and the MODELBEAST operator. Textures remain the one open item.
Also records the decimation table, because the floor is a real constraint and not
obvious: 500k faces is effectively lossless (IoU 0.965) but ~214k cannot be beaten,
and reaching it costs fidelity (0.823). Cause is the ~180k boundary edges the dual
grid emits for open surfaces - quadric decimation will not collapse them at any
setting, and o_voxel's remesh ran >20min on 214k faces before being killed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
base_color 0:3, metallic 3:4, roughness 4:5, alpha 5:6 - the pipeline's own
pbr_attr_layout. o_voxel's baker indexes this dict BY NAME and raises KeyError deep
inside to_glb on any missing slot rather than at the call, so a partial layout looks
like a baker bug. Needed by the texture stage; recorded now while it is in hand.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The raw cascade output is ~4M verts / 8M faces and is not usable as-is. cleanup.py is
ordinary mesh hygiene, kept out of the model code, and the ORDER is the whole point:
weld -> strip floaters -> decimate -> strip again -> fix normals
CORRECTION TO THE FLOATER COUNT. The health pass reported 52,855 components with
52,838 fragments under 100 faces, and I took those for stray shells. They were mostly
NOT: the decoder emits per-voxel vertices, so coincident corners are duplicated and
the same continuous surface reads as tens of thousands of islands. Welding FIRST
collapses it to a single component, and only 6,332 faces are genuinely stray. Ordering
the pass the other way round removes 250k faces of real geometry and calls it cleaning.
Two performance fixes, both because an operator runs this every job:
- Component labelling is a scipy union-find over the VERTEX graph, not
trimesh.face_adjacency. Same answer, ~240s -> ~1s on this mesh.
- fix_normals runs LAST, on the decimated mesh. It walks face adjacency, so on the
raw 7.99M-face mesh it costs minutes and the result is then thrown away by
decimation. Cleanup went ~243s -> ~20s.
Decimation is iterative. A single fast_simplification call will not reduce past
roughly 4.4% of its input whatever target_reduction (or agg) is asked for: from 7.99M
faces, targets of 200k, 50k and 20k ALL returned 351,535. Repeated smaller passes get
further because each re-evaluates quadrics on the collapsed mesh.
MEASURED, on the sample:
target 500,000 -> 499,984 faces silhouette IoU 0.965 19.3s
target 200,000 -> 214,322 faces silhouette IoU 0.823 34.8s
target 100,000 -> 214,322 faces silhouette IoU 0.823
target 20,000 -> 214,322 faces silhouette IoU 0.823
HONEST LIMITATION: ~214k is a hard floor, and reaching it costs real fidelity
(0.965 -> 0.823). The cause is the ~180,000 BOUNDARY edges the Flexible Dual Grid
produces for open surfaces - quadric decimation will not collapse those, and no
aggressiveness setting changes it. Below ~214k needs a REMESH, not a decimator.
500k is effectively lossless and is the setting to use; anything under 214k is not
currently reachable and the loop stops rather than spinning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
image_to_mesh() now runs the real cascade, not the single-stage shortcut:
structure 3048 voxels @32^3 (64^3 occupancy, MAX-POOLED DOWN)
LR SLAT 3048 x 32 shape_512 extractor
refine 13147 coords @64^3 four decoder stages -> coords -> quantise
HR SLAT 13147 x 32 shape_1024 extractor
mesh 3988052 verts, 7996876 faces @1024^3
TOTAL 258.4s, peak 27.9GB with every model resident silhouette IoU 0.969
Three things the cascade needed:
1. occupied_coords_at() - ss_dec always decodes 64^3 but the cascade STARTS at 32^3.
Upstream max-pools the boolean grid down by the ratio (a voxel survives if ANY of
its eight children was occupied). I had been feeding the raw 64^3 set to the HR flow.
2. decoder.upsample() - pushes the LR latent four stages in and returns COORDS, not
features. The predicted subdivisions grow the occupied set; those coords quantise
onto the HR flow's grid. Stops BEFORE stage `upsample_times`, as upstream does;
one stage further doubles the resolution and misplaces every voxel.
3. grid_resolution override on ProjConditioner - upstream backs the HR grid off in
128-unit steps while the token count exceeds max_num_tokens, so a dense object
degrades instead of exploding. refine_coords() implements that loop.
I WAS WRONG ABOUT THE HALO. The previous commit blamed the single-stage shortcut for a
0.639 silhouette IoU and predicted the cascade would fix it. The cascade measured
0.640 - no change. The real fault was in my VERIFICATION, not the pipeline: o_voxel
returns vertices in the voxel-grid frame, while ProjGrid rotates its lattice by
_BLENDER_ROT before projecting. Rotating the mesh the same way scores 0.969 on the
same geometry the earlier commit had already produced. Added mesh.to_camera_frame()
so the trap is named where it bites; the earlier mesh was correct all along.
The cascade is still the right thing - it is the shipped path, and staged loading
halves peak memory (12.8GB vs 22.6GB) when models are released between stages.
Also adds models.load_all(), so a server builds all five models plus both conditioners
ONCE. Warmup is ~71s against ~17s of compute, so an operator must never fork per job.
Holding everything resident costs 27.9GB peak - nothing on a 256GB box.
scripts/image_to_mesh.py exits non-zero if IoU < 0.85: a run that completes with a bad
reconstruction has failed even though nothing raised.
27/27 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The whole geometry chain now runs on a real photograph:
[1] occupancy 12948 voxels 19.7s
[2] cond proj (1, 262144, 2048) @ 64^3 2.6s
gathered proj (12948, 2048)
[3] SLAT (12948, 32) 88.7s
[4] MESH 3556515 verts, 7071196 faces 11.0s grid 1024^3
peak 22.6 GB, bounds inside the unit cube
Two bugs fixed on the way:
1. "global" must be FLAT [M,C] for the sparse blocks, not the dense stage's [B,T,C].
The sparse cross-attention takes a token stack plus an explicit layout, so the
dense shape dies inside to_kv's reshape rather than anywhere informative. Gathering
now reshapes it, and refuses batch > 1 rather than silently mislabelling a layout.
2. o_voxel needs the decoder OUTPUT grid, not its configured resolution. The shape
decoder applies four 2x upsamples, so a res-64 latent decodes into 1024^3, while
the config says 256 (upstream overrides it per run via set_resolution). Passing 256
raised an opaque out-of-bounds inside o_voxel's hashmap insert. Added
output_resolution() and a guard that names the real cause.
HONEST LIMITATION - this is NOT yet the shipped cascade. Upstream's
sample_shape_slat_cascade runs the 512 flow (res 32) first, denormalises, UPSAMPLES
THE COORDINATE SET through the shape decoder, then runs the 1024 flow on the refined
coords. Running the HR flow straight off the 64^3 occupancy set yields a complete,
exportable mesh whose silhouette IoU is 0.639 - against 0.842 for the occupancy grid
that seeded it. The gap is a halo of geometry outside the true silhouette, exactly
what the missing coordinate refinement would prune. Do not read the current mesh
quality as the model's; wiring the cascade is the next step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shape decoder's 7 channels are not an SDF — O-Voxel solves a QEF over a Flexible
Dual Grid, which is what lets it carry open and non-manifold surfaces:
0:3 vertex offset in-voxel, (1+2m)*sigmoid(v)-m so it may sit OUTSIDE its own cell
3:6 per-axis intersection logits, thresholded at 0
6:7 quad split weight through softplus
mesh.py is the MLX->torch boundary for export. o_voxel's convert/postprocess are
native (C++/Metal) and deliberately NOT ported: o-voxel builds a CPU CppExtension when
CUDA is absent, and the trellis-2 lane on this fleet already runs it with a Metal
baker, so reusing that build beats reimplementing a QEF solver in MLX. Installed into
the shared venv from ~/Documents/trellis-2-mrp-mlx/o-voxel; it needs cv2 and xatlas,
and NOT utils3d (which drags in open3d, with no cp312 wheel).
Verified against the REAL shape_dec (292/292 params, resolution 256):
decoded 5954 voxels x 7ch -> 5954 vertices, 6886 faces -> GLB written
Not watertight, correctly: the input was a random latent, and FlexiDualGrid represents
open surfaces by design. Vertices land inside the octant of the unit cube matching the
sparse coords fed in, which is the check that the grid indexing is right.
Still to wire: the SLAT stage itself (sparse latents seeded from the occupancy coords,
with proj features gathered at those coords).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the conditioning. shape_512 / shape_1024 / tex_1024 run a second HIGH-RES
branch — NAF upsamples the DINOv3 patch map to 512/1024 guided by the RGB image, the
proj grid samples that too, and the branches concatenate. That is why those stages
have proj_channels = embed_dim*2 (2048).
CORRECTS AN EARLIER CLAIM: I said natten is never imported and can be skipped. True of
Pixal3D's own source — but natten is a dependency of NAF (valeoai/NAF), which arrives
at RUNTIME via torch.hub and is not vendored. That is what README Step 3 is for. The
warning was real; the reason was one level down.
natten is a dead end on Apple Silicon regardless:
cutlass-fna requires libnatten, which the arm64 build does not produce
flex-fna CPU only ('not on a CUDA, ROCm, or CPU device: mps'), AND refuses
different head dims for QK vs V — which is exactly NAF's shape
(qk=64, v=256). Worked around, CPU took 243s at 256px and was
OOM-KILLED (exit 137) at the 512 the pipeline actually needs.
REPLACED BY AN EXACT REDUCTION, not an approximation. NAF resizes K/V from the 32x32
patch map with nearest-exact and then dilates by exactly the upsample factor, so the
dilated high-res neighborhood samples one position per low-res cell and collapses to a
plain clamped 9x9 neighborhood on the 32x32 grid, shared by every high-res pixel in
that cell. Verified against natten's own kernel at three dilations:
LR 16 -> HR 64 (dil 4) max diff 7.153e-07
LR 16 -> HR 128 (dil 8) max diff 7.153e-07
LR 32 -> HR 256 (dil 8) max diff 7.153e-07 (float32 epsilon)
Grouping queries by low-res cell also avoids materialising the high-res neighborhood,
which would be ~87GB of gathered V at 512. Result, on MPS:
32 -> 512 2.4s (natten: OOM-killed)
64 -> 512 0.2s
64 -> 1024 0.9s
Also fixes a caching bug that stranded NAF on whichever device loaded first, and one
in my own wiring: the high-res branch must reuse the SAME ProjGrid at image_size, not
a new one at naf_target_size. The normalised coordinate carries a 1/resolution term,
so the latter lands ~0.001 off in [-1,1] — a sub-pixel shift on every voxel, in the
one model whose entire premise is pixel alignment.
27/27 green (15 proj incl. the NAF stage vs upstream at corr 1.00000000, 5 sampler,
5 naf vs natten, 2 decoders).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
image_to_occupancy() runs the structure stage on an actual photo: preprocess ->
DINOv3 -> proj back-projection -> ss_flow -> ss_dec -> 64^3 occupancy.
VERIFICATION THAT MATTERS: scripts/run_structure.py re-projects the occupied voxels
through the same camera and compares against the input alpha matte. On the upstream
sample that is silhouette IoU 0.842 with 12948 voxels occupied (4.94% of 64^3). This
is the model's own headline claim, so it is the right thing to assert — 'it ran
without crashing' would pass just as happily on a generic blob.
Two real bugs this phase found, neither visible without reading the shipped configs:
1. THE SAMPLER WAS MISSING guidance_rescale. The checkpoint's own pipeline.json sets
0.7 for the structure stage and 0.5 for shape_slat, so this fires at the model's
DEFAULT settings — omitting it silently overcooks every structure prediction. Now
implemented (Lin et al. CFG rescale) and diffed against upstream's
ClassifierFreeGuidanceSamplerMixin, run directly rather than reimplemented.
2. The sampler defaults were wrong: the real ss stage is steps=12 / rescale_t=5.0 /
guidance 7.5 / interval [0.6,1.0], not the steps=25 / rescale_t=3.0 the smoke test
assumed. All three stages' real params now live in pipeline.py, read from
pipeline.json rather than guessed.
TIMINGS, measured with interleaved reps after warmup (the first pass attributed the
same 11s of residual warmup to both 'rescale' and 'torch contention'; it was neither):
cold run 89.3s
warm, full settings 16.5s
warm, CFG off 9.2s -> CFG costs 1.80x, as expected for 10/12
steps falling inside the guidance interval
guidance_rescale ~0s -> free
torch/MPS contention ~0s -> DINOv3 can stay resident
peak memory 6.8GB
THE FINDING THAT SHAPES THE OPERATOR: warmup is ~71s against ~17s of actual compute,
i.e. 4x the work. A MODELBEAST operator MUST hold the models resident across jobs
rather than fork per job — the trellis2 lane shows the same shape (47.9s cold vs 2.5s
warm pipeline_load). Cost this in before optimising any kernel.
17/17 tests green (12 proj + 5 sampler).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pixel-aligned conditioning is the ONLY thing separating this port from the
trellis2_mlx operator already in MODELBEAST — upstream's main branch is the
TRELLIS.2 backbone, so everything else here is TRELLIS.2 with a different head.
This lands that head.
proj.py ProjGrid, project_points, bilinear_sample, distance_from_fov — MLX
dino.py DINOv3 ViT-L/16 left in torch on MPS (run once per image, outside the
25-step loop; transformers gives exact parity for free)
cond.py encode_image_proj equivalent -> {'global','proj'} + zero uncond
The extractor has no sparse conv, so upstream RUNS on CPU torch here and is a real
oracle. All 12 checks diff against it, not against a transcription:
bilinear_sample vs grid_sample max diff 2.4e-07 corr 1.00000000
project_points pixels/depth/mask exact
ProjGrid forward (ss, 16^3) max diff 1.9e-05 corr 1.00000000
extractor global tokens max diff 0.0e+00 corr 1.00000000
extractor proj features max diff 4.8e-06 corr 1.00000000
Three details that a plain transcription gets wrong and eyeballing cannot catch:
grid_sample's align_corners=False maps a normalised coord to ((c+1)*size-1)/2, not
(c+1)/2*(size-1) — half a texel, invisible until you compare; padding_mode='border'
clamps the SOURCE INDEX before corners are taken, not the corners after, which
changes the weights on every silhouette edge (tested with deliberately out-of-range
grid coords); and the camera looks down -Z, so a sign slip still yields a plausible
grid that samples the mirror image.
Also corrects a shape assumption from the earlier smoke test: 'global' is CLS + 4
register tokens = [B,5,1024], NOT the 1370 image tokens. The patch tokens go to the
proj branch. That asymmetry IS the architecture.
Note the parameterless final layer_norm in extract_features — not model.norm, which
has weights. Same trap as the ss_flow bug: no checkpoint trace, 200x output error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The structure stage now runs end to end on Metal: noise -> ss_flow (12-step Euler)
-> latent -> ss_dec -> 64^3 occupancy grid. 759 ms/step for the 1.3B DiT, 50 ms for
the decoder. Output lands at 1.36% occupancy, which is the right order for a surface
in a 64^3 grid.
Sampler details worth recording, both from upstream:
- CFG is a LERP (g*pos + (1-g)*neg), NOT neg + g*(pos-neg). Those differ non-linearly
in strength rather than failing outright, so it is silent when wrong. Tested.
- Guidance interval forces strength to 1 outside its window, which halves model calls
there: 6 calls for 4 steps rather than 8. Tested by counting.
Schedule matches upstream to 1e-16 and constant velocity integrates exactly at any
step count.
Remaining for a real image->3D run is CONDITIONING, not models: DINOv3 features plus
Pixal3D's camera back-projection for the view-aligned 'proj' half. Deliberately not
porting DINOv3 - it is a stock ViT run once per image, outside the denoising loop, so
torch on MPS is the right tool and transformers gives exact parity for free.
All seven Pixal3D models now load and run. shape_dec 292/292, tex_dec 284/284, both
with zero missing/unmapped/mismatched keys - the 8-param difference between them is
exactly the four to_subdiv layers, since tex_dec has pred_subdiv=False.
Behaviour is right: 12 latent voxels grow SELECTIVELY through four stages
(12 -> 91 -> 193 -> 358 -> 1150) rather than x8 each time, which would have reached
49,152. Scale lands at 1/16. shape_dec emits the hardcoded 7 channels and its vertex
head produces offsets inside the [-0.5, 1.5] band its sigmoid+voxel_margin allows.
tex_dec, guided by shape_dec's masks, reproduces exactly the same voxel count.
VERIFICATION CAVEAT, recorded in the README: these two are the only models using sparse
conv, so upstream cannot run here and there is no numerical oracle. Unlike the five
models verified at correlation 1.0, these are checked structurally and behaviourally
only. Weaker evidence, and labelled as such rather than presented alongside the
verified results.
74/74 params, max abs diff 4.6e-4 on values around -163 (~3e-6 relative), 60ms for a
16^3 latent -> 64^3 occupancy grid.
Fully dense, so no sparse ops involved - but MLX's Conv3d is channels-LAST where torch
is NCDHW, so tensors are carried channels-last throughout and transposed only at the
boundaries. The converter already emits [O,kz,ky,kx,I] to match. pixel_shuffle_3d had
to be rewritten for that layout: the (H,s)(W,s)(D,s) interleave order is what matters
and getting it wrong scrambles the grid while preserving its shape.
This completes the structure stage end to end: image -> ss_flow -> latent -> ss_dec.
slat_flow joins ss_flow: max abs diff 9.3e-6, 700/700 params, against upstream running
on CPU torch. All three SLAT checkpoints load clean and run (img2shape 512/1024 and
imgshape2tex, the last taking 64 in-channels since shape is concatenated).
The SLAT flows differ from ss_flow in two ways, both handled in the shared core:
tokens are a SparseTensor's voxels so attention runs per batch item, and RoPE phases
are NOT shipped - positions are the input's own coordinates, so they are derived at
call time by rope_phases_from_coords.
tests/oracle_slat.py keeps the CPU-torch oracle harness: it patches both the dense and
sparse flash-attn kernels with SDPA equivalents. Use it rather than reasoning about
correctness - it has already caught three bugs a perfect 700/700 key match did not.
700/700 params, max abs diff 1.2e-5 on the real 1.3B checkpoint.
Key realisation: the flow models have no sparse conv, so upstream RUNS on CPU torch
with flash-attn swapped for SDPA. That gives a real numerical oracle - unavailable for
the sparse path, where spconv cannot be installed at all.
It was needed. Three bugs survived a loader reporting a perfect 700/700 with zero
missing and zero unmapped keys:
- a parameterless final LayerNorm (no params -> no checkpoint trace) that the output
was 200x too large without
- rope_phases being complex64, so the rotation is a complex multiply
- qk_rms_norm belonging before rope rather than after
Weight-key matching is necessary but nowhere near sufficient for a port.
Converter is light because Pixal3D ships safetensors with sibling .json configs, so
there is no architecture to infer and no pickle to unpack. Only rank-5 tensors are
rearranged; dtype is preserved (upcasting fp16->fp32 doubled 24GB for no benefit) and
the KRSC remap is verified a pure permutation of values.
Measured: the four flow models (~20GB) have ZERO 5-D tensors - pure transformers that
never touch sparse conv. shape_dec/tex_dec carry 40 KRSC kernels each, ss_dec 20 dense
Conv3d, and classify_5d separates them correctly on the real weights.
Corrects the earlier 'gap is two ops' claim: that was right about modules/sparse but
undercounted the model blocks. The configs show all four flow models need RoPE,
qk_rms_norm and AdaLN modulation, and the decoders need SparseConvNeXtBlock3d and
SparseResBlockC2S3d.
Every SparseConv3d in the Pixal3D model is (channels, out, 3) - stride=1/padding=None,
so spconv dispatches to SubMConv3d, which trellis_sparse_mlx already implements and
benchmarks. attn_mode is 'full' throughout. The container split (VarLenTensor /
SparseTensor) and the get/register_spatial_cache spelling are already in the shared core.
Remaining gap is two ops: SparseUpsample and SparseSpatial2Channel, both cache-paired
with a matching downsample rather than recomputing structure.
