image_to_occupancy() runs the structure stage on an actual photo: preprocess ->
DINOv3 -> proj back-projection -> ss_flow -> ss_dec -> 64^3 occupancy.
VERIFICATION THAT MATTERS: scripts/run_structure.py re-projects the occupied voxels
through the same camera and compares against the input alpha matte. On the upstream
sample that is silhouette IoU 0.842 with 12948 voxels occupied (4.94% of 64^3). This
is the model's own headline claim, so it is the right thing to assert — 'it ran
without crashing' would pass just as happily on a generic blob.
Two real bugs this phase found, neither visible without reading the shipped configs:
1. THE SAMPLER WAS MISSING guidance_rescale. The checkpoint's own pipeline.json sets
0.7 for the structure stage and 0.5 for shape_slat, so this fires at the model's
DEFAULT settings — omitting it silently overcooks every structure prediction. Now
implemented (Lin et al. CFG rescale) and diffed against upstream's
ClassifierFreeGuidanceSamplerMixin, run directly rather than reimplemented.
2. The sampler defaults were wrong: the real ss stage is steps=12 / rescale_t=5.0 /
guidance 7.5 / interval [0.6,1.0], not the steps=25 / rescale_t=3.0 the smoke test
assumed. All three stages' real params now live in pipeline.py, read from
pipeline.json rather than guessed.
TIMINGS, measured with interleaved reps after warmup (the first pass attributed the
same 11s of residual warmup to both 'rescale' and 'torch contention'; it was neither):
cold run 89.3s
warm, full settings 16.5s
warm, CFG off 9.2s -> CFG costs 1.80x, as expected for 10/12
steps falling inside the guidance interval
guidance_rescale ~0s -> free
torch/MPS contention ~0s -> DINOv3 can stay resident
peak memory 6.8GB
THE FINDING THAT SHAPES THE OPERATOR: warmup is ~71s against ~17s of actual compute,
i.e. 4x the work. A MODELBEAST operator MUST hold the models resident across jobs
rather than fork per job — the trellis2 lane shows the same shape (47.9s cold vs 2.5s
warm pipeline_load). Cost this in before optimising any kernel.
17/17 tests green (12 proj + 5 sampler).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The structure stage now runs end to end on Metal: noise -> ss_flow (12-step Euler)
-> latent -> ss_dec -> 64^3 occupancy grid. 759 ms/step for the 1.3B DiT, 50 ms for
the decoder. Output lands at 1.36% occupancy, which is the right order for a surface
in a 64^3 grid.
Sampler details worth recording, both from upstream:
- CFG is a LERP (g*pos + (1-g)*neg), NOT neg + g*(pos-neg). Those differ non-linearly
in strength rather than failing outright, so it is silent when wrong. Tested.
- Guidance interval forces strength to 1 outside its window, which halves model calls
there: 6 calls for 4 steps rather than 8. Tested by counting.
Schedule matches upstream to 1e-16 and constant velocity integrates exactly at any
step count.
Remaining for a real image->3D run is CONDITIONING, not models: DINOv3 features plus
Pixal3D's camera back-projection for the view-aligned 'proj' half. Deliberately not
porting DINOv3 - it is a stock ViT run once per image, outside the denoising loop, so
torch on MPS is the right tool and transformers gives exact parity for free.
