bf8b1a35d5
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
m3ultra
|
b6f0d76619 |
Texture stage: tex SLAT + PBR decode, and the bake order that makes it tractable
The texture flow is imgshape2tex - it denoises 32 PBR channels while SEEING the shape latent, so in_channels is 64 against out_channels 32. Upstream feeds the shape latent as concat_cond and the model does sparse_cat([x, concat_cond], dim=-1); both share coords, so it reduces to a channel concat. Added to slat_flow and carried on the sampler (it is fixed for the whole trajectory and must reach BOTH CFG branches). Verified running on the real checkpoints: 3,988,052 PBR voxels x 6 channels in 65.8s (base_color 0:3, metallic 3:4, roughness 4:5, alpha 5:6). Two things here fail SILENTLY rather than loudly, so both are asserted in comments: 1. shape_slat arrives DENORMALISED - the shape stage un-standardises it for the decoder - but the texture flow was trained against the standardised form. It is re-normalised before use as concat_cond. Skipping that gives a plausible mesh with wrong colours, not an error. 2. tex_dec has pred_subdiv=False: it cannot invent subdivisions and must be handed the shape decoder's subs as guides, so texture voxels land on the geometry that was actually built. The decoder's output is mapped * 0.5 + 0.5 into [0,1], the range o_voxel expects. BAKE ORDER. Handing o_voxel the raw ~8M-face mesh hangs - the same wall the standalone remesh test hit (killed at 20min), and the trellis2 lane's own operator note says the uncapped bake peaks at 75GB. So the mesh is welded, stripped of floaters and decimated BEFORE baking; the baker samples the attribute VOLUME at mesh positions, so a decimated mesh still gets correct colours. Measured on the way through: welded 3,988,052 -> 3,983,672 verts floaters 12 components -> 1 kept, 6,332 faces dropped decimated 7,996,876 -> 214,322 faces pre-bake 34.6s That floater count is worth noting: 12 components, not the 52,855 the first health pass reported. Welding first is what makes the difference. remesh now defaults OFF in to_glb, unlike upstream. Upstream runs on CUDA; this is the CPU/Metal build and its remesher took >20 minutes on a 214k-face mesh. It is also handed an already-clean mesh, so there is far less for it to fix. Operator gains texture + texture_size params; geometry-only stays the default because it is ~3min against the textured path's extra flow and bake. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
m3ultra
|
79ef81988c |
Real image -> occupancy grid, with a silhouette check and honest timings
image_to_occupancy() runs the structure stage on an actual photo: preprocess ->
DINOv3 -> proj back-projection -> ss_flow -> ss_dec -> 64^3 occupancy.
VERIFICATION THAT MATTERS: scripts/run_structure.py re-projects the occupied voxels
through the same camera and compares against the input alpha matte. On the upstream
sample that is silhouette IoU 0.842 with 12948 voxels occupied (4.94% of 64^3). This
is the model's own headline claim, so it is the right thing to assert — 'it ran
without crashing' would pass just as happily on a generic blob.
Two real bugs this phase found, neither visible without reading the shipped configs:
1. THE SAMPLER WAS MISSING guidance_rescale. The checkpoint's own pipeline.json sets
0.7 for the structure stage and 0.5 for shape_slat, so this fires at the model's
DEFAULT settings — omitting it silently overcooks every structure prediction. Now
implemented (Lin et al. CFG rescale) and diffed against upstream's
ClassifierFreeGuidanceSamplerMixin, run directly rather than reimplemented.
2. The sampler defaults were wrong: the real ss stage is steps=12 / rescale_t=5.0 /
guidance 7.5 / interval [0.6,1.0], not the steps=25 / rescale_t=3.0 the smoke test
assumed. All three stages' real params now live in pipeline.py, read from
pipeline.json rather than guessed.
TIMINGS, measured with interleaved reps after warmup (the first pass attributed the
same 11s of residual warmup to both 'rescale' and 'torch contention'; it was neither):
cold run 89.3s
warm, full settings 16.5s
warm, CFG off 9.2s -> CFG costs 1.80x, as expected for 10/12
steps falling inside the guidance interval
guidance_rescale ~0s -> free
torch/MPS contention ~0s -> DINOv3 can stay resident
peak memory 6.8GB
THE FINDING THAT SHAPES THE OPERATOR: warmup is ~71s against ~17s of actual compute,
i.e. 4x the work. A MODELBEAST operator MUST hold the models resident across jobs
rather than fork per job — the trellis2 lane shows the same shape (47.9s cold vs 2.5s
warm pipeline_load). Cost this in before optimising any kernel.
17/17 tests green (12 proj + 5 sampler).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
John
|
d43a6dfd69 |
Flow Euler sampler + structure-stage pipeline
The structure stage now runs end to end on Metal: noise -> ss_flow (12-step Euler) -> latent -> ss_dec -> 64^3 occupancy grid. 759 ms/step for the 1.3B DiT, 50 ms for the decoder. Output lands at 1.36% occupancy, which is the right order for a surface in a 64^3 grid. Sampler details worth recording, both from upstream: - CFG is a LERP (g*pos + (1-g)*neg), NOT neg + g*(pos-neg). Those differ non-linearly in strength rather than failing outright, so it is silent when wrong. Tested. - Guidance interval forces strength to 1 outside its window, which halves model calls there: 6 calls for 4 steps rather than 8. Tested by counting. Schedule matches upstream to 1e-16 and constant velocity integrates exactly at any step count. Remaining for a real image->3D run is CONDITIONING, not models: DINOv3 features plus Pixal3D's camera back-projection for the view-aligned 'proj' half. Deliberately not porting DINOv3 - it is a stock ViT run once per image, outside the denoising loop, so torch on MPS is the right tool and transformers gives exact parity for free. |