docs: FABLE_STARTUP_PACKET §7 — round-2 verification (ADOPT confirmed HIGH, transpose policy verified, parity oracle CPU-runnable, BiRefNet fetched; flags ss_dec double-transpose + bf16-crash + downcast in convert_weights)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
m3ultra
parent
abfa344565
commit
3b6b9124f9
@ -191,3 +191,173 @@ Invocation (later, GPU): `TRELLIS2_DUMP_DIR=~/Documents/trellis2-bench/parity_fi
|
|||||||
7. **Before any commercial preprocessing run:** `huggingface-cli download ZhengPeng7/BiRefNet` (§4 TODO 1) — otherwise only the non-commercial RMBG-2.0 path works.
|
7. **Before any commercial preprocessing run:** `huggingface-cli download ZhengPeng7/BiRefNet` (§4 TODO 1) — otherwise only the non-commercial RMBG-2.0 path works.
|
||||||
|
|
||||||
**Prime directives:** parity-first (every kernel/edit ships a ≤1e-4-rel PT-parity test + a BENCHMARKS.md row); do NOT edit shared `mlx_arsenal` in place (§3 Target 3); default BiRefNet for game-bound assets (§4). The GPU A0 gen is the ONLY job that must wait for a free box.
|
**Prime directives:** parity-first (every kernel/edit ships a ≤1e-4-rel PT-parity test + a BENCHMARKS.md row); do NOT edit shared `mlx_arsenal` in place (§3 Target 3); default BiRefNet for game-bound assets (§4). The GPU A0 gen is the ONLY job that must wait for a free box.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7 · PREP ROUND 2 — VERIFICATION RESULTS (2026-07-19)
|
||||||
|
|
||||||
|
> Round-2 deep-verify of §1–§6 against **live source + on-disk safetensors headers**. Still
|
||||||
|
> inspection-only: **no GPU gen, no weights loaded, no prod code touched** (`py_compile` /
|
||||||
|
> header-parse / `pytest --collect-only`). Where §7 says CONFIRMED / CORRECTION / CONTRADICTION,
|
||||||
|
> **trust §7 over the original section.**
|
||||||
|
|
||||||
|
### 7.1 · ADOPT confidence — §2 stance → **CONFIRMED (HIGH)**
|
||||||
|
|
||||||
|
- **Clone DONE & pinnable.** `~/Documents/trellis2-mlx-staging/trellis2-jourloy/`, HEAD `754d403`
|
||||||
|
(2026-07-17, Jourloy — matches "PR#175"), branch `main` **is** the PR head, tree clean.
|
||||||
|
**Pin `754d403`** when forking.
|
||||||
|
- **Superset CONFIRMED — stronger than §2 claimed.** All 4 named artifacts present
|
||||||
|
(`tests/test_mlx_parity.py`, `trellis2/backends.py`, `scripts/generate_asset.py`,
|
||||||
|
`requirements_macos.txt`) **plus 4 extra test files** (`test_backend_fallbacks`,
|
||||||
|
`test_background_preprocess`, `test_generate_asset`, `test_model_revisions`) **and** a new
|
||||||
|
`trellis2/model_revisions.py` (HF revision pins). `mlx_backend` = 3172 LOC / 16 files, all
|
||||||
|
`py_compile` clean.
|
||||||
|
- **`mlx_backend` numerics = `trellis2-apple` byte-for-byte.** `diff -rq`: 14/16 files identical;
|
||||||
|
the only 2 that differ (`dinov3.py`, `pipeline.py`) add **HF revision-pinning / offline kwargs
|
||||||
|
only — zero algorithm change.** §2's "identical mlx_backend" holds for numerics; jourloy is a
|
||||||
|
strict superset (adds reproducibility plumbing on top of the same math).
|
||||||
|
- **`--backend mlx-experimental` cleanly wired** (`generate_asset.py:333`→`create_mlx_pipeline`,
|
||||||
|
`:343`) behind a real **Metal-completeness probe gate** (`:170-187` → `backends.probe_metal_backends()`)
|
||||||
|
that refuses a half-built stack instead of silently degrading. MPS stays the E2E path.
|
||||||
|
- **§2's 4 UNVERIFIED smells — now resolved:**
|
||||||
|
|
||||||
|
| # | smell | round-2 disposition |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | structure_decoder no-op remap + dict-child conv load | **BENIGN by construction** (leaves pre-alloc'd `mx.zeros`; MLX param tree recurses dict/list; `load_weights(strict=True)` replaces leaves). Residual = exact key-name match only, **not** exercised by parity test → run a 2-sec non-GPU load smoke on ss_dec. |
|
||||||
|
| 2 | vae_decoders softplus/quad_lerp | **BENIGN** — it's the numerically-stable softplus identity `max(x,0)+log1p(exp(-|x|))` = torch softplus. Real line `vae_decoders.py:196` (§2's `:1204` misattributed). Needs one decode-boundary fixture (E1/E2). |
|
||||||
|
| 3 | non-windowed global attention | **NEEDS-RUNTIME-CHECK / POTENTIALLY REAL — highest-value item.** `_sdpa` does full global SDPA, **no window mask**. If upstream sparse flow is windowed → numeric divergence **and** O(N²) blowup at 1024-cascade. Tiny SDPA parity test validates the primitive, not the windowing semantics. **Verify vs upstream sparse-attn config before trusting shape/tex output.** |
|
||||||
|
| 4 | "custom Metal kernel" docstrings | **CONFIRMED BENIGN (cosmetic)** — grep `metal_kernel|@mx.custom|.metal|kernel_source` over `mlx_backend/` = zero matches; code calls MLX built-in `mx.fast.scaled_dot_product_attention`. No hand-written kernel anywhere. |
|
||||||
|
|
||||||
|
- **Bonus (confirms §2):** sparse-conv perf crux is **REAL** — `sparse_conv.py` is naive
|
||||||
|
gather→matmul→sum, 512 MB-chunked with a per-chunk `mx.eval` host sync, O(batch·D³) dense LUT,
|
||||||
|
**no `flex_gemm`.** Matches the ~10× claim.
|
||||||
|
- **Confidence HIGH, not lowered:** nothing contradicted the ADOPT case; several findings
|
||||||
|
strengthened it. Two items still need a *runtime* signal (smell 3 windowed-attn, smell 1
|
||||||
|
dict-child load) — **both cheap, non-GPU; run before the A0 gate** (see 7.6).
|
||||||
|
|
||||||
|
### 7.2 · WEIGHT-CONVERT — transpose policy **VERIFIED vs disk**; converter has 2 bugs + 1 double-transpose contradiction
|
||||||
|
|
||||||
|
All 8 modules parsed **header-only** (no tensor bytes). Tensor counts match §5 **verbatim**
|
||||||
|
(640/640/640/640/640/292/284/74). Classifier found **ZERO ambiguous 5D tensors** — conventions
|
||||||
|
separate cleanly by shape.
|
||||||
|
|
||||||
|
| module(s) | dtype on disk | 5D convs | §5 policy | matches disk |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 5 DiTs | **BF16** ×640 ea | 0 (215 Linear + 120 RMSNorm γ + 305 1D) | passthrough + dtype cast | ✅ |
|
||||||
|
| shape_dec / tex_dec | **F16** | 40 sparse ea, `(Co,K,K,K,Ci)` | **NO transpose** | ✅ |
|
||||||
|
| ss_dec | **F32×38 + F16×36** (2 F32 convs = `input_layer` + `out_layer.2`) | 20 dense, `(Co,Ci,K,K,K)` | **transpose `(0,2,3,4,1)`** | ✅ |
|
||||||
|
|
||||||
|
Transpose/convention classification in §5 holds on disk with **zero contradictions**. The
|
||||||
|
converter's `else: raise RuntimeError` on any unclassified 5D fails **loud**, so no silent
|
||||||
|
transpose-convention corruption is possible. **But three items must be fixed before any convert run:**
|
||||||
|
|
||||||
|
- 🔴 **BUG 1 (BLOCKING)** `convert_weights.py:59` `v.numpy()` on `torch.bfloat16` →
|
||||||
|
`TypeError: unsupported ScalarType BFloat16`. **Crashes on all 5 BF16 DiTs** (the bulk of the
|
||||||
|
pipeline); only survives on F16/F32 decoders. Loud, not silent. Fix: `v.float().numpy()`, or
|
||||||
|
bit-reinterpret via `uint16`, or drop torch and use `mx.load`/`mx.save_safetensors`.
|
||||||
|
- 🟠 **BUG 2 (SILENT precision loss)** uniform `target_dtype` default `"bf16"` (`:46,81-83,94`)
|
||||||
|
downcasts the decoders under default args: shape_dec/tex_dec **F16→bf16** (lose 3 mantissa bits,
|
||||||
|
strictly worse + ~20% slower on M1); ss_dec's **2 F32 convs → bf16** (lose 16 bits on the most
|
||||||
|
sensitive first/last SDF layers). **Contradicts §5's own dtype column (decoders = F16).** Fix:
|
||||||
|
per-file dtype policy — DiTs→bf16 (fp16 on M1), decoders→preserve on-disk dtype.
|
||||||
|
- ⚠️ **CONTRADICTION — ss_dec DOUBLE-TRANSPOSE (silent-corruption risk).** §5/converter transpose
|
||||||
|
ss_dec `(0,2,3,4,1)` **at load**. But the live MLX `structure_decoder.py` stores PT-format
|
||||||
|
`(Co,Ci,kD,kH,kW)` **verbatim** and transposes **at forward-call time** (`conv3d`,
|
||||||
|
`structure_decoder.py:14`). If the converter *also* transposes ss_dec, the decoder
|
||||||
|
**double-transposes → garbage, silently.** **Resolve before converting:** convert ss_dec to
|
||||||
|
PT-format on disk (NO load transpose) to match this file, OR move the transpose — pick exactly
|
||||||
|
one, not both.
|
||||||
|
- **Port dependency (not the converter):** the MLX sparse-conv module MUST do the
|
||||||
|
`(Co,K,K,K,Ci)→(Kvol,Ci,Co)` reshape at load (`conv_none.py:106`) or the `*_next_dc` decoders
|
||||||
|
emit garbage. Converter correctly leaves this to the module.
|
||||||
|
|
||||||
|
### 7.3 · PARITY HOOKS — all 8 targets match live source, **ZERO corrections to the hook**
|
||||||
|
|
||||||
|
`parity_dump_hook.py` needs no edits. Source `trellis2_image_to_3d.py` is frozen at upstream
|
||||||
|
`75fbf01` (2026-06-05) + 1 benign local Mac cuda-guard (`:590-591`, downstream of every hook →
|
||||||
|
shifts no fixture line). **No post-packet drift.** Verified def/return/call (corrections to §5's
|
||||||
|
fixture table where it drifted):
|
||||||
|
|
||||||
|
| fixture | def | return | call-site(s) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| get_cond (A/A′) | 164 | **181** (no-neg) / **183-186** (dict) | 539, 540 |
|
||||||
|
| sample_sparse_structure (B) | 188 | 235 | 542 |
|
||||||
|
| sample_shape_slat (C, non-cascade) | 237 | 275 | 547, 557 |
|
||||||
|
| sample_shape_slat_cascade (C′) | 277 | 364 (tuple `slat,hr_res`; `transform` takes `r[0]`) | 567, 579 |
|
||||||
|
| sample_tex_slat (D) | 391 | 432 | 551, 561, 574, 586 |
|
||||||
|
| decode_shape_slat (E1) | 366 | 389 | 470 |
|
||||||
|
| decode_tex_slat (E2) | 434 | 453 | 471 |
|
||||||
|
| decode_latent (E3) | 456 (`@torch.no_grad` :455) | 486 | 592 |
|
||||||
|
|
||||||
|
- **§5 CORRECTION:** the fixture table cites get_cond "return **:177**" — that line is the model
|
||||||
|
call `cond = self.image_cond_model(image)`, **not** the return. Real returns are **:181 / :183-186**.
|
||||||
|
Cosmetic (hook wraps by method name), but corrected.
|
||||||
|
- **Normalization gotcha CONFIRMED:** `shape_slat.pt` / `tex_slat.pt` captured **POST-denorm**
|
||||||
|
(`:271-273`, `:360-362`, `:428-430`); tex sampler **re-normalizes** shape_slat back down at
|
||||||
|
`:407-409` before `concat_cond` (:419). Port must replicate **both** the on-return de-norm and the
|
||||||
|
denorm→renorm round-trip or fixtures diverge by a constant affine (silent ≤1e-4 failure).
|
||||||
|
- **UNCERTAIN — flag for exec session:** the hook's `__main__` `runpy`-delegates to vendor
|
||||||
|
`generate.py`; that file's existence + CLI contract (`--pipeline-type` / `--seed` /
|
||||||
|
`--texture-size`) was **not** checked. Verify before the dump run — a missing/renamed delegate
|
||||||
|
means the hook installs but never fires.
|
||||||
|
|
||||||
|
### 7.4 · PREPROCESSING — BiRefNet **PRESENT**; D2 **IMPLEMENTED**
|
||||||
|
|
||||||
|
- **BiRefNet fetched (§1 "NOT present" TODO → DONE).** Cache:
|
||||||
|
`/Users/m3ultra/.cache/huggingface/hub/models--ZhengPeng7--BiRefNet/snapshots/e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4/`.
|
||||||
|
Size **424 MB** (`model.safetensors` = 444,473,596 B) — **§1/§4's "~900 MB" was high.**
|
||||||
|
License **MIT** (README `license: mit`) — **§1/§4's "Apache-2.0" label was wrong.** Ungated, no login.
|
||||||
|
- **BiRefNet branch VERIFIED CORRECT (not stubbed):** RMBG-2.0 *is* a BiRefNet checkpoint (identical
|
||||||
|
`auto_map`/`architectures`); forward `model(inp)[-1].sigmoid()` holds (`birefnet.py:2087/2090`);
|
||||||
|
deps satisfied by the rmbg venv — **no cv2 needed** (`birefnet.py` has zero opencv refs). Only
|
||||||
|
metadata was hardened; no code change.
|
||||||
|
- **D2 (light normalization) IMPLEMENTED** behind `normalize_light=False` (default **OFF** → v0
|
||||||
|
silhouette stays A/B-identical to fal). New `_normalize_light` (`prep_local_draft.py:84`, PIL/numpy
|
||||||
|
only): 1/99-pct auto-levels (flat-channel guarded) + gray-world WB; wired at step **1b** (after
|
||||||
|
upscale, before matte — honors "matte LAST"). CLI `--normalize-light`. Functional smoke passed on a
|
||||||
|
synthetic cast image.
|
||||||
|
- **Draft status:** `~/Documents/trellis2-mlx-staging/prep_local_draft.py` now 236 lines,
|
||||||
|
`py_compile` OK + imports clean under rmbg venv. Signature
|
||||||
|
`prep_local(in_path, out_path, commercial=False, normalize_light=False)`. **Staging only — still
|
||||||
|
NOT wired into prod** (§4.5 wire-in + tunable A/B still open; need a GPU-free box for real A/B).
|
||||||
|
|
||||||
|
### 7.5 · MLX TEST VENV — oracle is **one command from running**
|
||||||
|
|
||||||
|
- **Venv READY:** `/Users/m3ultra/Documents/trellis2-mlx-staging/mlx-test-venv` (CPython **3.12.13**,
|
||||||
|
uv). `mlx==0.32.0` (+`mlx-metal` 0.32.0), `torch==2.13.0` (prebuilt wheel, no compile),
|
||||||
|
`numpy` 2.5.1, `pytest` 9.1.1. (safetensors/transformers/einops also installed but the oracle uses
|
||||||
|
**none** of them — droppable for a leaner venv.)
|
||||||
|
- **Collects clean:** `--collect-only` → **4 tests collected, 0 import errors, 0 skips** (0 skips
|
||||||
|
proves both `importorskip("mlx.core")` and `importorskip("torch")` succeeded on the box). **Not yet
|
||||||
|
executed** — a real pass is still pending (low-risk caveat: `np.asarray(mlx_array)` under numpy
|
||||||
|
2.5.1 surfaces only on the real run, not collection).
|
||||||
|
- **GPU/Metal?** 4 tests run microsecond ops on tiny tensors. MLX's default device is Metal (GPU), so
|
||||||
|
they *touch* the GPU — but this is trivial library compute, **NOT a generation/inference job**, uses
|
||||||
|
**no Metal build packages** (mtlgemm/mtldiffrast/cumesh) and no compile. Won't meaningfully contend
|
||||||
|
for the box; force CPU with `mx.set_default_device(mx.cpu)` if ever desired.
|
||||||
|
- **EXACT run command** (use the venv python — a bare `pytest` uses the wrong interpreter, correcting
|
||||||
|
§6/§2):
|
||||||
|
```
|
||||||
|
ssh -n m3ultra@100.89.131.57 'cd ~/Documents/trellis2-mlx-staging/trellis2-jourloy && \
|
||||||
|
../mlx-test-venv/bin/python -m pytest tests/test_mlx_parity.py -v'
|
||||||
|
```
|
||||||
|
Expected: **4 passed.** Baked tolerances: LayerNorm `2e-5`, RoPE `1e-6`, SDPA `3e-5`,
|
||||||
|
SparseConv `2e-5` (all tighter than the ≤1e-4 target).
|
||||||
|
|
||||||
|
### 7.6 · UPDATED FIRST MOVES — deltas to §6
|
||||||
|
|
||||||
|
- **§6 step 2 (clone Jourloy): DONE** — `trellis2-jourloy/` @ `754d403`, tree clean. Skip the clone;
|
||||||
|
just `git checkout 754d403` / pin it when forking.
|
||||||
|
- **§6 step 3 (run parity oracle): venv READY, oracle collects clean.** Now a one-command move — run
|
||||||
|
the exact command in **7.5**. **Do NOT use the bare `pytest tests/...` from §6/§2** (wrong
|
||||||
|
interpreter).
|
||||||
|
- **§6 step 7 / §4 TODO 1 (BiRefNet fetch): DONE** — skip. Present at the 7.4 path (424 MB, MIT).
|
||||||
|
- **NEW — before any weight-convert run (§5):** fix converter **Bug 1** (blocking bf16 crash) +
|
||||||
|
**Bug 2** (silent decoder downcast), AND resolve the **ss_dec double-transpose** contradiction (7.2).
|
||||||
|
Else DiT convert crashes, decoders lose precision, or ss_dec silently corrupts.
|
||||||
|
- **NEW — before the A0 GPU gate:** run the two cheap **non-GPU** runtime checks — (a) ss_dec
|
||||||
|
dict-child weight-load smoke, and especially **(c) global-vs-windowed attention vs the upstream
|
||||||
|
sparse-attn config** (the one smell that can be a true correctness bug, not just slow). Both fit
|
||||||
|
CPU/inspection while the box is busy.
|
||||||
|
- **Unchanged:** §6 step 5 (single A0 GPU gate — seed 42, 1024-cascade, compare 124.8 s) is still the
|
||||||
|
**only** job that must wait for a free box; §6 step 6 (Hunyuan Target 1) still safe in parallel.
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user