PROCITY/E-progress.md
m3ultra 23c0667dca Lane E round 9 (E1+E2): VRAM/texture audit + half-res book/toy atlases
E1 memory audit: measured all texture sources (decoded RGBA+mips). Resident VRAM
~240MB default boot / ~420MB worst case (all flags + all shop types); stock packs
the swing. Cheap win: half-res'd book+toy atlases (quick pull-cards, not the deep
record dig — covers verified legible) → stock VRAM 323->181MB, no index/UV change,
re-published to depot. Verdict: fine to ship v2.0; flagged stockpack cache-cap (C)
+ KTX2/Basis (4-8x) for v3. E2 hygiene: validate --depot 0 err, pack-QA 3/3,
provenance-drift clean (27 live), 0 orphan atlases. Numbers + verdict in AUDIT.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 10:59:22 +10:00

23 KiB
Raw Blame History

LANE E — ASSETS · progress (PROCITY-E)

2026-07-14. Owner: PROCITY-E. Scope: pipeline/*, web/assets/*, web/assets/manifest.json. Reviewed against docs/LANES/LANE_E_ASSETS.md + docs/LANES/ROUND3_INSTRUCTIONS.md. Nothing in other lanes' files was touched.

Round 9 (Fable) — VRAM audit + depot hygiene (ship-round DD)

  • E1 — VRAM/texture audit: DONE + committed. Measured every texture source (decoded RGBA+mips). Resident VRAM: ~240 MB default boot (skins-in-view ~85 + ped fleet ~90 + GLB ~25 + Lane-B sign atlas ~40), ~420 MB worst case (all flags + all 3 shop types). Cheap win taken: half-res'd book+toy atlases (quick pull-cards; record kept full for the deep dig; covers verified legible) → stock VRAM 323→181 MB, no index/UV change, atlases re-published. Verdict: fine to ship v2.0 — no compression/lazy-load needed for the tag. Flagged for v3: stockpack cache-cap (C) + KTX2/Basis.
  • E2 — depot/manifest hygiene: confirmed green. validate --depot 0 err · pack-QA 3/3 · provenance -drift clean (_published.json == 27 live) · 15 stock atlases live, 0 orphans.
  • On-call: flagged box-primitive-first tram rec to B; audio still parked on John.

Round 8 (Fable) — pack kitchen (book/toy) + pack-QA gate

  • E1 — pack pipeline generalized + book/toy sample packs. build_stock_pack.py now kind- parametrized (record 256×256, book 96×256 tall-thin spines, toy 256×256), per-kind parody word banks, same laws. Book/toy sample packs built + staged (?localdepot) + indexes committed to pipeline/packs/ so Lane C wires book/toy shelves now. Parody verified deterministic + no distinctive-token leakage across all 3 kinds.
  • E2 — pack-QA gate: DONE. validate_pack.py (wired into validate_manifest.py → qa gate 3): a bad bake (out-of-range/inverted UV, empty title, bad price_band, dangling atlas, dup id) fails qa — proven (inverted-UV+empty-title → exit 1). qa --strict GREEN 5/5.
  • Tags pushed (John-delegated): origin/main = HEAD (1a20501) + v1.0/v1.1/v2.0-alpha/v2.0-beta all live on origin — verified.
  • REAL packs LIVE (John unblocked DB access). All 3 packs rebuilt from GODVERSE (m1ultra Postgres clones, read-only SELECT, peer auth): record 350 (discogs_full.inventory — John's real store stock + real Discogs covers), book 311 + toy 273 (dealgod.products by category, real product photos + prices). Metadata parody-transformed (no real trademarks); covers are real art (decision #2); book prices seeded (info-only source). 15 atlases published to the depot, indexes committed to pipeline/packs/ + staged, all pass the pack-QA gate. build_stock_pack.py gained --from-tsv (extract on the DB box → build here) + portrait book cell + seeded-price fallback. Access path: deployment DB was Docker-internal/SSH-denied → used the m1ultra clones per John. No fal spend.
  • On-call (B rain-streak/puddle, D door-swing tell): none requested.

Round 7 (Fable) — the two feeders (ped merge + stock pack)

  • E1 — ped sub-mesh merge: DONE + committed (447188a, ref lane-e/round7-peds). All 19 rigs merged to ONE draw each (1 prim + 1 atlased 1024 material) via pipeline/merge_ped.py (Blender join → UV-pack → EMIT-bake all materials into one atlas). Skinning + 65-joint skeleton preserved, walk/idle untouched. Fleet 92→19 draws; near-cap 116→24 (79% cut). Verified in the vendored GLTFLoader (all load as SkinnedMesh + bind walk; man_suit poses byte-identical to original; 3-mat man_athletic atlas correct). Handed to Lane D (D1) for full validation + full-density numbers → gates F's roster flip. qa --strict GREEN 5/5.
  • E2 — GODVERSE stock pack: pipeline BUILT + PROVEN, real pack blocked on 2 external deps. pipeline/build_stock_pack.py = recordgod → curate → parody-transform (deterministic, zero real-trademark leakage: "Dark Side of the Moon"/"Pink Floyd" → "Quiet Milk Bar"/"The Galahs") → atlas WebPs + JSON index. Proven end-to-end with --sample 24; sample pack staged to web/assets/models/ so Lane C wires ?stock=real against the real index format now. Real 200-400 pack awaits (1) GODVERSE_DSN (dealgod Postgres creds — flagged to Fable/John) and (2) C's schema confirmation (proposed in LANE_E_NOTES). One command (--from-db --count 300) once they land.
  • E3 — on-call (B rain-streak/puddle texture): none requested. No fal spend (balance ~$21.36).

Round 6 (Fable) — non-destructive pipeline + on-call

  • E1 — publish.py non-destructive + provenance-drift gate: DONE + committed.
    • publish.py now merges into _published.json (read existing → union → sorted write), never replaces — fixes the R5 --only clobber (record dropped 27→1). Proven: a real no-op re-publish of record_crate kept the record at 27 (was 27), not 1.
    • Post-publish + a new --verify mode drift-check the record against the live depot (/api/list filtered to procity_*) and warn on either direction. --verify confirms 27 == 27.
    • validate_manifest.py gains a provenance-drift gate: every manifest procity_* GLB must be in _published.json, else QA fails. Proven: simulating the R5 clobber (record→1) makes the validator exit 1 with per-file drift errors; restored → green. So the clobber class fails loudly.
    • Record verified: _published.json (27) is byte-exact vs the live depot — Fable's R5 regeneration was correct, no fix needed.
    • qa.sh --strict GREEN (5 gates incl. F1's new v2-flags harness).
  • On-call for B/C (task 3): available for bus_shelter/bin scale-origin tweaks (B) and glass_case/magazine_rack in-room issues (C). Broken-prop fal re-gen pre-authorized ≤2 attempts (decision #5) — none needed yet. No speculative generation.

Round 5 (Fable) — hero-prop bake + fal fallbacks + depot growth

  • E1 — hero-prop bake-to-lowpoly: DONE + committed. New pipeline/bake_lowpoly.py (Blender headless): voxel-remesh → decimate → Cycles base-colour bake onto fresh UVs (R3's lesson: simplify-only floors on shell-soup; the re-bake is the point). All 6 heavy TRELLIS props ≤8k tris (drinks_fridge 77k→8k, milkshake_mixer 73k→8k, listening_booth 54k→8k, counter_till 32k→8k, novelty_record 26k→8k, arcade_cabinet 25k→8k). Fleet 288k→48k tris, 18M→5.2M. Each eyeballed (docs/shots/laneE/props_baked.png — fridge needed a coarser voxel re-bake, rest clean)
    • verified loading in the vendored GLTFLoader. Re-published; validate --depot 0 err. Originals gitignored in pipeline/.props_orig/. → Lane C re-measures the record interior (was ~273k).
  • E3 — fal.ai fallbacks: DONE. glass_case + bus_shelter + magazine_rack via fal Hunyuan3D v2 (pipeline/gen_props_fal.py, key from MODELBEAST settings). Frosted/opaque concepts (flux_local) gave the reconstructor a surface — all 3 succeeded where TRELLIS failed. Baked to ≤8k (EMIT albedo bake — Hunyuan materials need base-colour→emission or DIFFUSE bakes near-black), verified loading, published, validate --depot 0 err. Total fal spend ~$0.64 (glass 1 att, bus 2 att, mag 1 att × ~$0.16), well under the <$1 estimate; guardrail held. Costs logged in AUDIT R5. Last two primitive-only fittings (glass_case, magazine_rack) retired; new street bus_shelter. Manifest now 17 fittings + 9 furniture (26 GLBs). Contact sheet docs/shots/laneE/props_fal.png.
  • E2 — depot growth: DONE (light, consumer-driven). The v2 features are procedural (dig uses generated sleeves, B's window shader + D's roster need no GLBs), so per Fable "no unconsumed entries" I added two clear-consumer props: street/bin (council-bin → Lane B) + fit/crate_stack (milk-crate-stack → Lane C floor piles/dig). Both 3k tris, re-normalized + published, verified loading, validate --depot 0 err. Also resolved Lane B's B3 ask — novelty_record 26.5k→8k via the E1 bake. Manifest now 18 fittings + 10 furniture (28 GLBs). Further growth on request.

Round 4 (Fable) — ped decimation (gate-3 critical path) + asset fixes

  • E1 — ped decimation: DONE + committed (518678d, ref lane-e/round4-peds). The 19 ped rigs in web/models/peds/ were 924k tris total (31k56k each) — the gate-3 blocker. Decimated to ≤3k each (15642805; fleet 924k→46.5k, ~2.45k avg) via gltf-transform weld → simplify --ratio 0.05 --error 0.12 (meshoptimizer — works on these manifold skinned meshes, unlike R3 TRELLIS shell-soup). Skinning + skeleton fully preserved (65 joints intact, walk/idle untouched); contact sheet docs/shots/laneE/peds_decimated.png clean at street distance. Originals gitignored in pipeline/.peds_orig/; per-ped in pipeline/_peds_decim.json. → Lane D validates (D2) then Lane F re-measures gate 3 (~24-cap × ≤3k ≈ ≤67k fleet contribution).
  • E2a — record_crate FIXED, verified + published. Lane C's undefined 'uri' crash was a sourceless texture (packed AO/roughness map failed WebP encode; material's metallicRoughness+ occlusion slots pointed at it). New tool pipeline/fix_glb_textures.py strips those slots (three.js loads textures lazily → no crash). Verified it loads in the real vendored GLTFLoader (meshes=1, tris=5000). Re-published to the depot (the fixed version is live, tex refs all-sourced).
  • E2b — counter-with-till (~2 m): LIVE. New counter_till fitting (procity_fit_counter_till_01.glb, 1.61 × 0.64 m, 1.15 m tall, 32k tris) — timber counter + cabinet doors + period till. Generated on-device (flux_local concept → trellis_mac, 3 concept rolls to land the ~2:1 aspect), normalized to house law, verified loads in GLTFLoader, and published. Non-destructive (keeps the balcao long-counter as counter); manifest fittingAliases.countercounter_till; Lane C re-maps kind countercounter_till in glb.js.
  • Publish: DONE (John approved). Both re-exports live; manifest validate --depot GREEN (0 err, 23 GLBs live). publish.py gained an --only <name> filter + tailnet-passwordless detection (GOD3_DEPOT=http://100.94.195.115:8788, decision #4).
  • Not doing (per Fable decisions): fal.ai spend (glass-case/bus-shelter parked); hero-prop bake-to-lowpoly.

Round 3 (Fable) — hero props on MODELBEAST + local-depot mode

  • Hero props, free & on-device (pipeline/gen_props.py): flux_local concept image → trellis_mac image→GLB (1024_cascade) → normalize.py → thumbs → manifest. 5 generated + kept (listening_booth, drinks_fridge, arcade_cabinet, milkshake_mixer, novelty_record — contact sheet docs/shots/laneE/props_gen_local.png), cash_register reused from the depot (vintage-cash-register.glb). 3 didn't make it: magazine_rack → in-engine primitive (112k-tri wire, the thin-wire gotcha); glass_case + bus_shelterfal.ai fallback list (TRELLIS can't do transparent glass; sf3d gives a featureless blob) — gated on John, ≤$2.70, both have primitive fallbacks. Every kept thumbnail eyeballed; no legible branding (see AUDIT provenance).
  • Local-depot mode (core/loaders.js ?localdepot=1 + pipeline/stage_local_depot.py): resolves depot:<file> to web/assets/models/ served by the dev server, so Lanes B/C/F validate GLB loading with no network / before publish. Verified live in-browser (loads a depot: GLB, CDN path intact).
  • Manifest: 14 fittings + 8 furniture (5 hero + cash_register added); validate_manifest.py green (0 err). Provenance recorded in AUDIT.md (props are 100% synthetic/local — cleanest in the set).
  • Known tradeoff: TRELLIS meshes are dense shell-soup — decimate floors 2577k tris (heavier than the ≤5k library law). Accepted for focal/optional/LOD-able props; texture-bake-to-lowpoly is a follow-up.
  • Published (self-serve, John's go 2026-07-14): the 5 prop GLBs + thumbs pushed to the depot via the passwordless tailnet path (decision #4); all 22 GLBs re-verified live. qa.sh --strict GREEN (4/4 gates; manifest validator 0 err / 0 warn with the props live). Provenance was recorded first.
  • Not committed: this round is held for Fable review (per John) — all Lane E round-3 changes are in the working tree, uncommitted. GLB binaries stay git-ignored/staged; the depot has them.

Status at a glance

Step Deliverable State
1 Audit pipeline/AUDIT.md (+ glb_stat.py, _audit_stats.json) done
2 Normalize pipeline/normalize.py16 GLBs + 16 thumbs, verified done (local); publish gated on GOD3_PW
3 Manifest web/assets/manifest.json + validate_manifest.py (+ build_manifest.py) done, validates green
4 Facade skins gen_skins.py --local backend + 23-skin gap pack DONE — 23 skins generated on-device, free, harvested + mapped
5 Hero props round-3 gen_props.py (MODELBEAST, free) → 5 local + cash-register reuse DONE local (see Round 3 above; 2→fal, 1→primitive)
6 Local-depot loaders.js ?localdepot=1 + stage_local_depot.py DONE, verified in-browser
pipeline/README.md + contact sheets in docs/shots/laneE/ done

Step 4 no longer costs anything. The 23 facades were generated locally on the M3 Ultra via MODELBEAST flux_local (FLUX.2-klein-4B on MPS) — free, ~5.7s each, weights already cached. No OpenRouter key, no ~9¢ spend. See "Step 4 (done, local)" below.

Two things still need John (both optional — the game runs asset-free): (1) the depot password GOD3_PW to publish the 16 GLBs live; (2) a go/no-go on the paid hero-prop step (5 ≈ $2.70).

What landed

Step 1 — Audit

  • pipeline/glb_stat.py: dependency-free GLB parser (tris, bbox, texture px, Draco) — runs on ultra's py3.9. Full library stats → pipeline/_audit_stats.json.
  • pipeline/AUDIT.md: the library is a raw dump — almost nothing obeys the GLB law (only record_crate_plastic_01 was native-metres; scale/orientation arbitrary; tris 12→960k; textures to 4096; several branded/trademarked assets quarantined). Candidate→PROCITY-id table with status, coverage-vs-registry table, QUARANTINE list, depot cross-reference.
  • Toolchain reality documented (Blender present on ultra but off PATH; no coreutils/timeout).

Step 2 — Normalize & publish

  • pipeline/normalize.py: headless Blender, extends MESHGOD/finish_glb.py (solidify/decimate/ bright/base-origin) + adds height-normalize, outlier cleanup, texture→WebP≤1024, per-asset up/yaw, 256px thumbnail. Driven by pipeline/batch.json.
  • 16 assets normalized to house law (metres, +Y up, base origin, ≤5k tris, WebP ≤1024, no Draco). Every one eyeballed — Y-up confirmed by re-importing exports through render_glb.py. Footprints from Blender-space _normalize_results.json. Thumbs in web/assets/thumbs/.
    • fittings (9): record_crate, wire_shelf, clothes_rack, bookshelf, cube_shelf_wide, counter, coffee_table, work_table, box_crate
    • furniture (7): bench, bench_modern, bench_wood, longbench, park_bench, streetlight, food_cart
  • pipeline/publish.py: mirrors the meshgod→3GOD contract exactly (cookie hmac(GOD3_PW,"3god-ok"), POST /api/upload + /api/thumb). Reads the secret from env — never contains/logs it. --dry-run verified all 16 ready. Not published yet: GOD3_PW isn't set here. Run: export GOD3_PW=… && python3 pipeline/publish.py (idempotent, reversible).

Step 3 — Manifest

  • web/assets/manifest.json (v1): the contract Lanes B/C/D read. All 69 skins catalogued (25 facade / 10 sky / 10 ground / 8 wallpaper / 6 floor / 7 surface / 3 awning) + 16 fittings/ furniture with footprints, heights, thumbs. conventions block documents the GLB law, blank-signboard rule, corner-lot policy, fallbacks, and depot load path.
  • Every registry shop type has ≥2 facades (verified in the validator). Facades tagged signboard: blank|baked — 5 djsim/market/grimy skins carry baked sign text and must not get a name overlay; the other 20 are clean overlay targets.
  • pipeline/validate_manifest.py: Lane F gate — parses JSON, checks every skin exists, every GLB is local-or-on-depot, thumbs exist, footprints/heights sane, ≥2 facades/type. Green (16 local-only warnings until publish; --depot enforces live).
  • pipeline/build_manifest.py: reproducible generator (curated type-maps + derived skin/fitting lists) so the manifest regenerates when assets change.

Step 4 — Facade skin expansion (DONE, local)

  • pipeline/gen_skins.py: ported from thriftgod/gen_assets.pysame STYLE bible + blank- signboard template so new skins match. Resumable .genraw/, --dry-run/--only/--harvest.
  • --local backend (the one I used): MODELBEAST flux_local → FLUX.2-klein-4B on the M3 Ultra's MPS GPU. Apache/ungated, weights already cached (15 GB), free, ~5.7 s/img, one GPU job at a time. Direct subprocess to ~/Documents/MODELBEAST/server/operators/flux_local/run.py (--outdir/--params JSON) — no server, no cloud, no key. Cloud OpenRouter path kept as fallback.
  • All 23 generated → harvested → mapped. 0 failures, 2:11 total. Eyeballed every one (contact sheet docs/shots/laneE/facades_gen_local.png): every shopfront has a genuinely blank signboard (Lane B name-overlay target), style indistinguishable from the 25 anchors. The pack: toy (2), book (2), video (2), milkbar (3), wide dept anchors (2), stall/market (2), warehouse (2), arcade portal (1), residential house fronts (4), corner-lot -side walls (3).
  • Coverage after mapping (build_manifest.py FACADE_TYPES, mirrors registry.js + additive): record 3 · opshop 3 · toy 5 · book 5 · video 4 · pawn 7 · milkbar 5 · dept 4 · stall 4 — every registry type ≥3. stall went 2→4, retiring the Lane A single-facade gap.
  • Corner-lot -side skins now exist (brick/render/weatherboard): tagged face:"side" with empty types in the manifest so no shop-front pool ever picks a windowless wall; Lane B maps them to the exposed side face (conventions.corner_lots + new conventions.facade_face).
  • pipeline/SKIN_PROMPTS.md: identical pack kept for the free manual flow (fallback if MODELBEAST is ever down).

Step 5 — Bespoke hero props

  • pipeline/meshgod_batch.json + MESHGOD_BATCH.md: 9 solids (glass case, fridge, till, listening booth, arcade cabinet, magazine rack, milkshake mixer, novelty record, bus shelter) in the thriftgod_hero pattern. Documents the MeshGod gotcha (thin wires/flat discs → primitives), reuse-first (cash-register/glass-case likely already on depot), and the primitive fallbacks. Not run — gated on John (~$2.70). Nothing blocks: all have in-engine fallbacks.

Adversarial review & fixes (self-run, 4-dimension × verify)

Ran a 19-agent adversarial review over all deliverables (correctness / contract-fit / doc-accuracy / house-law), each finding independently verified. 15 confirmed (1 blocker, 5 major, 9 minor). All fixed and re-verified:

  • BLOCKER — off-origin exports. normalize.py joined meshes but never flattened parent node hierarchies, so assets under a parent empty shipped off-origin: park_bench was ~1 km off-origin, floating 30 m; food_cart 15 m off; clothes_rack sank 0.19 m. My earlier spot-check missed it (the clean assets have no hierarchy). Fixed: parent_clear(KEEP_TRANSFORM) + drop non-mesh nodes before join. Re-verified node-aware: all 16 now center=[0, h/2, 0], minY=0.0.
  • Major — tri budget counted faces, not triangles (latent for quad FBX/OBJ). Fixed: triangulate in clean_mesh so the ≤5k gate + reported counts are true triangles.
  • Major — fitting-id namespace. Added fittingAliases (registry semantic id → physical id) to the manifest. (Lane C's glb.js already maps its kinds to my physical ids, so it worked — this makes it explicit + future-proof.)
  • Major — facade map vs registry.js. Aligned my facade→type map to be a consistent superset of Lane A's registry.js; flagged that registry's stall pool has only 1 facade (Lane A gap).
  • Major — facing -Z not enforced. Corrected the docs: symmetric props are orientation-agnostic; no auto-facing (per-asset yaw available). Removed the false "enforced" claim.
  • Major — licence provenance. Strengthened the publish-set caveat (record per-asset provenance before pushing to the shared CDN) — a reason publish stays gated on John.
  • Minors: hardened glb_stat.py (node-aware world bounds + WebP size sniff — the exact blind spots that hid the blocker); validate_manifest.py liveness now uses a Range GET not HEAD; Akai-MPC trademark flag; corrected audit counts (59 parsed .glb, rejects 8, food_cart tris).

Acceptance check (LANE_E)

  • AUDIT.md complete.
  • ≥15 normalized GLBs (16) with thumbnails — live on 3GOD pending publish (needs cred).
  • manifest.json validates green; fittings/furniture carry footprints for B/C to place untouched.
  • Every registry shop type has ≥2 facades — now ≥3 (48 facades total). New skins generated (not just planned) and eyeball-matched to the anchor style (facades_style_anchor.png = 25 originals; facades_gen_local.png = 23 new local skins; fittings_normalized.png = the 16 GLBs — all in docs/shots/laneE/).
  • One-page pipeline/README.md (re-run each step, costs, where creds live).

Notes for the other lanes

  • Lane B / C: read web/assets/manifest.json. fittings/furnituredepot:<file> via core/loaders.js (placeholder box on miss — the game runs asset-free). Footprints are [w,d] metres on the ground; heights in metres; all base-origin, +Y up, facing Z.
  • Lane B: overlay shop names only on facades with signboard:"blank". Wide dept-* anchors (when generated) want a wider UV plane. awning/ground/sky skins are yours; wall/interior are Lane C's.
  • Lane F: wire python3 pipeline/validate_manifest.py into the gate (add --depot post-publish).

Open questions for Fable/John

  1. Publish now? I have 16 verified GLBs staged in pipeline/_normalized/ ready to push to the shared depot. It's an outward-facing write to shared infra + needs GOD3_PW, so I left it for your go rather than assume it.
  2. Spend the ~9¢ on facades? Resolved — done for free on-device (MODELBEAST flux_local). Only the ~$2.70 hero-prop MeshGod batch (Step 5) still wants a go/no-go; all props have in-engine primitive fallbacks, so it's a polish upgrade, not a blocker. (Note: image→GLB is also local on this box via sf3d/trellis_mac, but those weights are HF-gated — a setup step before that path is free too.)
  3. A few library assets were near-misses (garage_shelves, radiogram, market_stall — see AUDIT rejects). Worth a manual Blender re-origin pass later, or leave to primitives? I left them out.