diff --git a/docs/FABLE_STARTUP_PACKET.md b/docs/FABLE_STARTUP_PACKET.md index 2a32d30..4b0f18a 100644 --- a/docs/FABLE_STARTUP_PACKET.md +++ b/docs/FABLE_STARTUP_PACKET.md @@ -191,3 +191,173 @@ Invocation (later, GPU): `TRELLIS2_DUMP_DIR=~/Documents/trellis2-bench/parity_fi 7. **Before any commercial preprocessing run:** `huggingface-cli download ZhengPeng7/BiRefNet` (§4 TODO 1) — otherwise only the non-commercial RMBG-2.0 path works. **Prime directives:** parity-first (every kernel/edit ships a ≤1e-4-rel PT-parity test + a BENCHMARKS.md row); do NOT edit shared `mlx_arsenal` in place (§3 Target 3); default BiRefNet for game-bound assets (§4). The GPU A0 gen is the ONLY job that must wait for a free box. + +--- + +## 7 · PREP ROUND 2 — VERIFICATION RESULTS (2026-07-19) + +> Round-2 deep-verify of §1–§6 against **live source + on-disk safetensors headers**. Still +> inspection-only: **no GPU gen, no weights loaded, no prod code touched** (`py_compile` / +> header-parse / `pytest --collect-only`). Where §7 says CONFIRMED / CORRECTION / CONTRADICTION, +> **trust §7 over the original section.** + +### 7.1 · ADOPT confidence — §2 stance → **CONFIRMED (HIGH)** + +- **Clone DONE & pinnable.** `~/Documents/trellis2-mlx-staging/trellis2-jourloy/`, HEAD `754d403` + (2026-07-17, Jourloy — matches "PR#175"), branch `main` **is** the PR head, tree clean. + **Pin `754d403`** when forking. +- **Superset CONFIRMED — stronger than §2 claimed.** All 4 named artifacts present + (`tests/test_mlx_parity.py`, `trellis2/backends.py`, `scripts/generate_asset.py`, + `requirements_macos.txt`) **plus 4 extra test files** (`test_backend_fallbacks`, + `test_background_preprocess`, `test_generate_asset`, `test_model_revisions`) **and** a new + `trellis2/model_revisions.py` (HF revision pins). `mlx_backend` = 3172 LOC / 16 files, all + `py_compile` clean. +- **`mlx_backend` numerics = `trellis2-apple` byte-for-byte.** `diff -rq`: 14/16 files identical; + the only 2 that differ (`dinov3.py`, `pipeline.py`) add **HF revision-pinning / offline kwargs + only — zero algorithm change.** §2's "identical mlx_backend" holds for numerics; jourloy is a + strict superset (adds reproducibility plumbing on top of the same math). +- **`--backend mlx-experimental` cleanly wired** (`generate_asset.py:333`→`create_mlx_pipeline`, + `:343`) behind a real **Metal-completeness probe gate** (`:170-187` → `backends.probe_metal_backends()`) + that refuses a half-built stack instead of silently degrading. MPS stays the E2E path. +- **§2's 4 UNVERIFIED smells — now resolved:** + + | # | smell | round-2 disposition | + |---|---|---| + | 1 | structure_decoder no-op remap + dict-child conv load | **BENIGN by construction** (leaves pre-alloc'd `mx.zeros`; MLX param tree recurses dict/list; `load_weights(strict=True)` replaces leaves). Residual = exact key-name match only, **not** exercised by parity test → run a 2-sec non-GPU load smoke on ss_dec. | + | 2 | vae_decoders softplus/quad_lerp | **BENIGN** — it's the numerically-stable softplus identity `max(x,0)+log1p(exp(-|x|))` = torch softplus. Real line `vae_decoders.py:196` (§2's `:1204` misattributed). Needs one decode-boundary fixture (E1/E2). | + | 3 | non-windowed global attention | **NEEDS-RUNTIME-CHECK / POTENTIALLY REAL — highest-value item.** `_sdpa` does full global SDPA, **no window mask**. If upstream sparse flow is windowed → numeric divergence **and** O(N²) blowup at 1024-cascade. Tiny SDPA parity test validates the primitive, not the windowing semantics. **Verify vs upstream sparse-attn config before trusting shape/tex output.** | + | 4 | "custom Metal kernel" docstrings | **CONFIRMED BENIGN (cosmetic)** — grep `metal_kernel|@mx.custom|.metal|kernel_source` over `mlx_backend/` = zero matches; code calls MLX built-in `mx.fast.scaled_dot_product_attention`. No hand-written kernel anywhere. | + +- **Bonus (confirms §2):** sparse-conv perf crux is **REAL** — `sparse_conv.py` is naive + gather→matmul→sum, 512 MB-chunked with a per-chunk `mx.eval` host sync, O(batch·D³) dense LUT, + **no `flex_gemm`.** Matches the ~10× claim. +- **Confidence HIGH, not lowered:** nothing contradicted the ADOPT case; several findings + strengthened it. Two items still need a *runtime* signal (smell 3 windowed-attn, smell 1 + dict-child load) — **both cheap, non-GPU; run before the A0 gate** (see 7.6). + +### 7.2 · WEIGHT-CONVERT — transpose policy **VERIFIED vs disk**; converter has 2 bugs + 1 double-transpose contradiction + +All 8 modules parsed **header-only** (no tensor bytes). Tensor counts match §5 **verbatim** +(640/640/640/640/640/292/284/74). Classifier found **ZERO ambiguous 5D tensors** — conventions +separate cleanly by shape. + +| module(s) | dtype on disk | 5D convs | §5 policy | matches disk | +|---|---|---|---|---| +| 5 DiTs | **BF16** ×640 ea | 0 (215 Linear + 120 RMSNorm γ + 305 1D) | passthrough + dtype cast | ✅ | +| shape_dec / tex_dec | **F16** | 40 sparse ea, `(Co,K,K,K,Ci)` | **NO transpose** | ✅ | +| ss_dec | **F32×38 + F16×36** (2 F32 convs = `input_layer` + `out_layer.2`) | 20 dense, `(Co,Ci,K,K,K)` | **transpose `(0,2,3,4,1)`** | ✅ | + +Transpose/convention classification in §5 holds on disk with **zero contradictions**. The +converter's `else: raise RuntimeError` on any unclassified 5D fails **loud**, so no silent +transpose-convention corruption is possible. **But three items must be fixed before any convert run:** + +- 🔴 **BUG 1 (BLOCKING)** `convert_weights.py:59` `v.numpy()` on `torch.bfloat16` → + `TypeError: unsupported ScalarType BFloat16`. **Crashes on all 5 BF16 DiTs** (the bulk of the + pipeline); only survives on F16/F32 decoders. Loud, not silent. Fix: `v.float().numpy()`, or + bit-reinterpret via `uint16`, or drop torch and use `mx.load`/`mx.save_safetensors`. +- 🟠 **BUG 2 (SILENT precision loss)** uniform `target_dtype` default `"bf16"` (`:46,81-83,94`) + downcasts the decoders under default args: shape_dec/tex_dec **F16→bf16** (lose 3 mantissa bits, + strictly worse + ~20% slower on M1); ss_dec's **2 F32 convs → bf16** (lose 16 bits on the most + sensitive first/last SDF layers). **Contradicts §5's own dtype column (decoders = F16).** Fix: + per-file dtype policy — DiTs→bf16 (fp16 on M1), decoders→preserve on-disk dtype. +- ⚠️ **CONTRADICTION — ss_dec DOUBLE-TRANSPOSE (silent-corruption risk).** §5/converter transpose + ss_dec `(0,2,3,4,1)` **at load**. But the live MLX `structure_decoder.py` stores PT-format + `(Co,Ci,kD,kH,kW)` **verbatim** and transposes **at forward-call time** (`conv3d`, + `structure_decoder.py:14`). If the converter *also* transposes ss_dec, the decoder + **double-transposes → garbage, silently.** **Resolve before converting:** convert ss_dec to + PT-format on disk (NO load transpose) to match this file, OR move the transpose — pick exactly + one, not both. +- **Port dependency (not the converter):** the MLX sparse-conv module MUST do the + `(Co,K,K,K,Ci)→(Kvol,Ci,Co)` reshape at load (`conv_none.py:106`) or the `*_next_dc` decoders + emit garbage. Converter correctly leaves this to the module. + +### 7.3 · PARITY HOOKS — all 8 targets match live source, **ZERO corrections to the hook** + +`parity_dump_hook.py` needs no edits. Source `trellis2_image_to_3d.py` is frozen at upstream +`75fbf01` (2026-06-05) + 1 benign local Mac cuda-guard (`:590-591`, downstream of every hook → +shifts no fixture line). **No post-packet drift.** Verified def/return/call (corrections to §5's +fixture table where it drifted): + +| fixture | def | return | call-site(s) | +|---|---|---|---| +| get_cond (A/A′) | 164 | **181** (no-neg) / **183-186** (dict) | 539, 540 | +| sample_sparse_structure (B) | 188 | 235 | 542 | +| sample_shape_slat (C, non-cascade) | 237 | 275 | 547, 557 | +| sample_shape_slat_cascade (C′) | 277 | 364 (tuple `slat,hr_res`; `transform` takes `r[0]`) | 567, 579 | +| sample_tex_slat (D) | 391 | 432 | 551, 561, 574, 586 | +| decode_shape_slat (E1) | 366 | 389 | 470 | +| decode_tex_slat (E2) | 434 | 453 | 471 | +| decode_latent (E3) | 456 (`@torch.no_grad` :455) | 486 | 592 | + +- **§5 CORRECTION:** the fixture table cites get_cond "return **:177**" — that line is the model + call `cond = self.image_cond_model(image)`, **not** the return. Real returns are **:181 / :183-186**. + Cosmetic (hook wraps by method name), but corrected. +- **Normalization gotcha CONFIRMED:** `shape_slat.pt` / `tex_slat.pt` captured **POST-denorm** + (`:271-273`, `:360-362`, `:428-430`); tex sampler **re-normalizes** shape_slat back down at + `:407-409` before `concat_cond` (:419). Port must replicate **both** the on-return de-norm and the + denorm→renorm round-trip or fixtures diverge by a constant affine (silent ≤1e-4 failure). +- **UNCERTAIN — flag for exec session:** the hook's `__main__` `runpy`-delegates to vendor + `generate.py`; that file's existence + CLI contract (`--pipeline-type` / `--seed` / + `--texture-size`) was **not** checked. Verify before the dump run — a missing/renamed delegate + means the hook installs but never fires. + +### 7.4 · PREPROCESSING — BiRefNet **PRESENT**; D2 **IMPLEMENTED** + +- **BiRefNet fetched (§1 "NOT present" TODO → DONE).** Cache: + `/Users/m3ultra/.cache/huggingface/hub/models--ZhengPeng7--BiRefNet/snapshots/e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4/`. + Size **424 MB** (`model.safetensors` = 444,473,596 B) — **§1/§4's "~900 MB" was high.** + License **MIT** (README `license: mit`) — **§1/§4's "Apache-2.0" label was wrong.** Ungated, no login. +- **BiRefNet branch VERIFIED CORRECT (not stubbed):** RMBG-2.0 *is* a BiRefNet checkpoint (identical + `auto_map`/`architectures`); forward `model(inp)[-1].sigmoid()` holds (`birefnet.py:2087/2090`); + deps satisfied by the rmbg venv — **no cv2 needed** (`birefnet.py` has zero opencv refs). Only + metadata was hardened; no code change. +- **D2 (light normalization) IMPLEMENTED** behind `normalize_light=False` (default **OFF** → v0 + silhouette stays A/B-identical to fal). New `_normalize_light` (`prep_local_draft.py:84`, PIL/numpy + only): 1/99-pct auto-levels (flat-channel guarded) + gray-world WB; wired at step **1b** (after + upscale, before matte — honors "matte LAST"). CLI `--normalize-light`. Functional smoke passed on a + synthetic cast image. +- **Draft status:** `~/Documents/trellis2-mlx-staging/prep_local_draft.py` now 236 lines, + `py_compile` OK + imports clean under rmbg venv. Signature + `prep_local(in_path, out_path, commercial=False, normalize_light=False)`. **Staging only — still + NOT wired into prod** (§4.5 wire-in + tunable A/B still open; need a GPU-free box for real A/B). + +### 7.5 · MLX TEST VENV — oracle is **one command from running** + +- **Venv READY:** `/Users/m3ultra/Documents/trellis2-mlx-staging/mlx-test-venv` (CPython **3.12.13**, + uv). `mlx==0.32.0` (+`mlx-metal` 0.32.0), `torch==2.13.0` (prebuilt wheel, no compile), + `numpy` 2.5.1, `pytest` 9.1.1. (safetensors/transformers/einops also installed but the oracle uses + **none** of them — droppable for a leaner venv.) +- **Collects clean:** `--collect-only` → **4 tests collected, 0 import errors, 0 skips** (0 skips + proves both `importorskip("mlx.core")` and `importorskip("torch")` succeeded on the box). **Not yet + executed** — a real pass is still pending (low-risk caveat: `np.asarray(mlx_array)` under numpy + 2.5.1 surfaces only on the real run, not collection). +- **GPU/Metal?** 4 tests run microsecond ops on tiny tensors. MLX's default device is Metal (GPU), so + they *touch* the GPU — but this is trivial library compute, **NOT a generation/inference job**, uses + **no Metal build packages** (mtlgemm/mtldiffrast/cumesh) and no compile. Won't meaningfully contend + for the box; force CPU with `mx.set_default_device(mx.cpu)` if ever desired. +- **EXACT run command** (use the venv python — a bare `pytest` uses the wrong interpreter, correcting + §6/§2): + ``` + ssh -n m3ultra@100.89.131.57 'cd ~/Documents/trellis2-mlx-staging/trellis2-jourloy && \ + ../mlx-test-venv/bin/python -m pytest tests/test_mlx_parity.py -v' + ``` + Expected: **4 passed.** Baked tolerances: LayerNorm `2e-5`, RoPE `1e-6`, SDPA `3e-5`, + SparseConv `2e-5` (all tighter than the ≤1e-4 target). + +### 7.6 · UPDATED FIRST MOVES — deltas to §6 + +- **§6 step 2 (clone Jourloy): DONE** — `trellis2-jourloy/` @ `754d403`, tree clean. Skip the clone; + just `git checkout 754d403` / pin it when forking. +- **§6 step 3 (run parity oracle): venv READY, oracle collects clean.** Now a one-command move — run + the exact command in **7.5**. **Do NOT use the bare `pytest tests/...` from §6/§2** (wrong + interpreter). +- **§6 step 7 / §4 TODO 1 (BiRefNet fetch): DONE** — skip. Present at the 7.4 path (424 MB, MIT). +- **NEW — before any weight-convert run (§5):** fix converter **Bug 1** (blocking bf16 crash) + + **Bug 2** (silent decoder downcast), AND resolve the **ss_dec double-transpose** contradiction (7.2). + Else DiT convert crashes, decoders lose precision, or ss_dec silently corrupts. +- **NEW — before the A0 GPU gate:** run the two cheap **non-GPU** runtime checks — (a) ss_dec + dict-child weight-load smoke, and especially **(c) global-vs-windowed attention vs the upstream + sparse-attn config** (the one smell that can be a true correctness bug, not just slow). Both fit + CPU/inspection while the box is busy. +- **Unchanged:** §6 step 5 (single A0 GPU gate — seed 42, 1024-cascade, compare 124.8 s) is still the + **only** job that must wait for a free box; §6 step 6 (Hunyuan Target 1) still safe in parallel.