PROCITY/pipeline

PROCITY pipeline (Lane E) — how to re-run everything

The asset pipeline: audit ultra's library → normalize the good bits to house GLB law → publish to the 3GOD depot → ship web/assets/manifest.json (the contract Lanes B/C/D read) → fill facade

• prop gaps. Everything is optional at runtime — the game runs asset-free; this lane makes it gorgeous, it doesn't make it work.

Where things live

what	where
ultra (library + Blender)	ssh johnking@100.91.239.7 — key auth, no coreutils/timeout, py3.9, Blender at /Applications/Blender.app/Contents/MacOS/Blender (not on PATH)
model library	ultra ~/Documents/3D=models/{shop-fittings,street-furniture,furniture,props-scenes}
3GOD depot	https://digalot.fyi/3god — GET /a/<f>, GET /api/list, POST /api/upload?name=<f>
OpenRouter key (skins)	env OPENROUTER_KEY or ~/Documents/dealgod/lore/keys.env (never committed)
depot password (publish)	env GOD3_PW (never committed; ask John) — cookie = hmac(GOD3_PW,"3god-ok")
Ollama (free LLM)	http://100.89.131.57:11434 (qwen3) — for names/lore/cataloguing, unused so far
MODELBEAST (free local gen)	this box, ~/Documents/MODELBEAST + its own queue on :8777. flux_local = FLUX.2-klein-4B text→image on MPS (the facade skins — ungated, weights cached). sf3d/trellis_mac = image→GLB, local + PROVEN, weights already cached — trellis_mac (TRELLIS.2 1024_cascade) went 15/15 on thriftgod's Tier-1 props (Lane G/ultra, R1); sf3d = seconds-fast draft. Also live on the queue: bg_remove_local, mflux_image_edit, seedvr2_upscale. (The old "HF-gated / setup needed" note here was stale — flagged by ultra, corrected R22.)

The steps (each is idempotent / resumable)

1. Audit — AUDIT.md (free)

scp pipeline/glb_stat.py johnking@100.91.239.7:/tmp/
ssh johnking@100.91.239.7 'cd ~/Documents/3D=models && python3 /tmp/glb_stat.py --json shop-fittings street-furniture furniture props-scenes' > pipeline/_audit_stats.json

glb_stat.py is a dependency-free GLB parser (tris, bbox, texture sizes, Draco flag). Findings + licensing triage are written up in AUDIT.md.

2. Normalize — normalize.py → 16 GLBs + thumbs (free, runs on ultra)

scp pipeline/normalize.py pipeline/batch.json johnking@100.91.239.7:/tmp/
BL=/Applications/Blender.app/Contents/MacOS/Blender
ssh johnking@100.91.239.7 "\$BL --background --python /tmp/normalize.py -- /tmp/batch.json /tmp/procity_norm"
scp johnking@100.91.239.7:/tmp/procity_norm/_results.json pipeline/_normalize_results.json
scp 'johnking@100.91.239.7:/tmp/procity_norm/thumbs/*.png' web/assets/thumbs/
scp 'johnking@100.91.239.7:/tmp/procity_norm/*.glb' pipeline/_normalized/

Enforces the house GLB law (metres, +Y up, base origin, ≤5k tris, WebP ≤1024, no Draco), renders a 256px thumbnail each. Extends MESHGOD/scripts/finish_glb.py (solidify/decimate/bright/base-origin)

• adds texture-webp, height-normalize, outlier cleanup, per-asset up/yaw. Edit batch.json to add assets; always eyeball the thumbnails — node-transform-distorted sources (see AUDIT rejects) look fine in-frame but have wrong footprints.

3. Manifest — build_manifest.py + validate_manifest.py (free)

python3 pipeline/build_manifest.py          # → web/assets/manifest.json
python3 pipeline/validate_manifest.py       # Lane F gate: 0 errors = green
python3 pipeline/validate_manifest.py --depot   # stricter: require GLBs live on the depot

build_manifest.py derives skins from web/assets/gen/ + fittings from _normalize_results.json; the curated facade→shop-type maps live at the top of that file (edit there when adding skins).

4. Publish — publish.py → 3GOD depot (the one outward-facing step)

python3 pipeline/publish.py --dry-run        # free: list what would upload
export GOD3_PW=...                           # ask John
python3 pipeline/publish.py                  # POST each GLB + thumb to the depot (idempotent)

5. Facade skin gaps — gen_skins.py --local (DONE, free, on-device)

The 23-skin gap pack is already generated + harvested + mapped. To re-run or extend:

python3 pipeline/gen_skins.py --local --dry-run   # free: list the pack, no GPU
python3 pipeline/gen_skins.py --local             # MODELBEAST flux2-klein-4b on MPS → .genraw/ (~5.7s/img)
python3 pipeline/gen_skins.py --harvest           # → web/assets/gen/facade-*.jpg, then re-run build_manifest.py

--local drives ~/Documents/MODELBEAST/server/operators/flux_local/run.py directly (no server, no key, weights cached). It runs one GPU job at a time. Add skins by editing PROMPTS in gen_skins.py and their shop-type map in build_manifest.py (FACADE_TYPES). Always eyeball .genraw/ and delete rejects before --harvest. Cloud fallback (drop --local): OpenRouter google/gemini-3.1-flash-image, ~$0.004/img, needs OPENROUTER_KEY. Manual flow: SKIN_PROMPTS.md.

6. Bespoke hero props — gen_props.py (MODELBEAST) (round-3: DONE, free, on-device)

The meshgod_batch.json gap list now goes through MODELBEAST instead of fal.ai — free, local:

python3 pipeline/gen_props.py --dry-run                 # list props + what's done
python3 pipeline/gen_props.py --concepts                # flux_local concept images → .genprops/*.png (eyeball!)
python3 pipeline/gen_props.py --mesh --only glass-case  # trellis_mac 1024_cascade image→GLB (~5-8 min)
python3 pipeline/gen_props.py --mesh                    # all props with a concept, no glb
# then normalize (uses local Blender), stage, catalogue:
BL=/Applications/Blender.app/Contents/MacOS/Blender
"$BL" --background --python pipeline/normalize.py -- pipeline/batch_props.json pipeline/_normalized web/assets/thumbs
cp pipeline/_normalized/_results.json pipeline/_props_results.json && python3 pipeline/build_manifest.py

5 shipped (arcade_cabinet, listening_booth, drinks_fridge, milkshake_mixer, novelty_record) + cash-register reused from depot. magazine-rack → primitive (thin-wire gotcha); glass-case/bus-shelter → fal.ai fallback (TRELLIS can't do transparent glass; sf3d --draft gives a featureless blob). TRELLIS meshes run heavy (25–77k tris) — see AUDIT provenance. Eyeball every thumbnail; ≤2 local attempts then fal (John's go).

⚠️ Finishing a trellis_mac mesh — ultra's post-mortem (Lane G, R1) — folded in verbatim

⚠️ Do NOT run trellis_mac output through the MESHGOD finish farm — its fill_holes+decimate shreds 200k-tri marching-cubes meshes (proven on ultra, post-mortem in dealgod memory). The working pipeline: raw GLB → npx @gltf-transform/cli optimize --compress draco --texture-compress webp --texture-size 1024 --simplify-error 0.01 → 0.4–1.8 MB at raw-quality look.

Credit: Lane G / ultra (G-progress.md), proven post-mortem — it cost them a shredded batch so it shouldn't cost us one. One amendment, per Fable's Ruling 1 (docs/LANES/FABLE_TO_LANE_G.md): NO-DRACO WINS — the depot speaks uncompressed GLB and PROCITY's loader stays decoder-free. So drop --compress draco; everything else in ultra's command stands (WebP ≤1024 is already our law):

npx @gltf-transform/cli optimize IN.glb OUT.glb \
  --texture-compress webp --texture-size 1024 --simplify-error 0.01     # NO --compress draco

Files land ~2–3× bigger than the Draco'd version and that's fine — the depot holds ONE law. Note our own in-house paths remain valid for their own cases and are NOT the finish farm: bake_lowpoly.py (voxel-remesh + bake, for shell-soup that must hit a low budget) and decimate_props.py (collapse, detail-preserving — see the R13 instrument diet).

7. Local-depot mode — loaders.js ?localdepot=1 + stage_local_depot.py (round-3: DONE)

Lets Lanes B/C/F validate GLB loading with no network / before publish:

python3 pipeline/stage_local_depot.py     # mirror _normalized/*.glb → web/assets/models/ (git-ignored)
# then load the game with ?localdepot=1 — depot:<file> resolves to assets/models/<file> instead of the CDN

8. Audio pack — gen_audio.py (round-11: DONE, free, on-device)

Tool choice (Fable asked E to pick + document): pure procedural synthesis in numpy, not a neural model. MODELBEAST has no audio operator, and for this pack synthesis wins outright — $0, no model download, deterministic/seeded (house law), perfectly seamless loops (neural gens don't loop), parody-safe by construction (every sample is an original oscillator/noise render — nothing to imitate), and precise ≤25 MB budget control. Pipeline: numpy synth → WAV → ffmpeg OGG/Opus (+ M4A/AAC).

PY=~/Documents/MODELBEAST/venvs/mflux/bin/python     # numpy lives in the mflux venv
$PY pipeline/gen_audio.py --dry-run                  # list the 23-asset pack + budget
$PY pipeline/gen_audio.py                            # render all → web/assets/audio/*.ogg + *.m4a
$PY pipeline/gen_audio.py --only music               # just the music beds
python3 pipeline/build_manifest.py                   # catalogue → manifest.json audio:{ambience,sfx,music}

23 assets, 8.1 MB shipped (opus+aac): 6 ambience beds (street day/night, rain, 3 roomtones), 13 sfx (doorbell, door, till, riffle, toast, tram bell/rumble, 6 footsteps), 4 instrumental 90s music loops. Loudness: beds/music loudnorm -16 LUFS, sfx peak-normalised. Ear-ball = spectrogram sheet docs/shots/laneE/audio_spectrograms.png + numeric loop-seam/peak check (I can't subjectively listen — final ear-check is John's/F's). Add assets by editing PACK in gen_audio.py + the AUDIO_* maps in build_manifest.py. WAV scratch pipeline/.genaudio/ is git-ignored; OGG/M4A ship committed.

9. Gig layer — round-12 (v3.0-alpha, ?gigs=1) (DONE, free, on-device)

Three asset streams for one-pub-end-to-end, all $0:

# instruments (5 GLBs): flux concept -> trellis -> normalize, via the --batch override
python3 pipeline/gen_props.py --batch pipeline/gig_instruments.json --concepts   # eyeball first
python3 pipeline/gen_props.py --batch pipeline/gig_instruments.json --mesh        # ~3-4 min each
BL=/Applications/Blender.app/Contents/MacOS/Blender
"$BL" --background --python pipeline/normalize.py -- pipeline/gig_norm_batch.json pipeline/_normalized web/assets/thumbs
# posters (4 skins): portrait flyers with a blank name-band (B/C overprint like signboards)
python3 pipeline/gen_skins.py --posters && python3 pipeline/gen_skins.py --harvest
# gig audio (3): pub-rock live bed + crowd-walla + applause — already in gen_audio.py PACK
~/Documents/MODELBEAST/venvs/mflux/bin/python pipeline/gen_audio.py --only music   # etc.
python3 pipeline/build_manifest.py     # catalogues skins.poster + audio gig assets + fittings

Instruments: amp/guitar/bass reconstruct well; thin objects (mic-stand pole, drum hardware) may fall to D's primitive (round-3 thin-wire lesson — spec allows "primitives when absent"). Posters carry nameZone [x,y,w,h] in skins.poster. Gig audio is flagged gig:true in manifest.audio.

Cost summary

step	cost
audit, normalize, manifest, validate, publish	free (compute + bandwidth)
facade skin gap-fill (23 imgs)	free — done on-device (MODELBEAST flux_local); cloud fallback ~9¢
hero props (round-3)	free — 5 via MODELBEAST trellis_mac; glass-case/bus-shelter on fal.ai list (≤$2.70, John's go)
audio pack (round-11, 23 assets)	free — 100% procedural synthesis (numpy → ffmpeg), $0, no model

Artifacts (not committed — regenerated from the steps)

pipeline/_normalized/ (staging GLBs, live on depot), pipeline/.genraw/ (raw skin gen). Committed: the scripts, *.md, batch.json, meshgod_batch.json, _audit_stats.json, _normalize_results.json, web/assets/manifest.json, web/assets/thumbs/, docs/shots/laneE/.

10. Motion library — clips_export.py → clips_pack.py → clips_verify.py (round-41 §41.1: DONE, free, on-device)

46 Mixamo clips → 6 grouped skeleton-only GLBs in web/models/clips/ (3 498 124 B, 0 tris) + a new web/assets/motion_manifest.json. Source is ultra's ~/Documents/MOTIONLIB/green/mixamo_full/ only (Mixamo = GREEN: usable in games, no attribution, never redistribute as clip packs). The selection is checked in at pipeline/clips_r41.json — every asset by name, no wildcard.

python3 - <<'EOF' > /tmp/fbx_list.txt      # 1. the 46 names
import json;print('\n'.join(c['file'] for g in json.load(open('pipeline/clips_r41.json'))['groups'].values() for c in g['clips']))
EOF
mkdir -p /tmp/fbx && ssh johnking@100.91.239.7 \
  "cd ~/Documents/MOTIONLIB/green/mixamo_full && tar czf - $(tr '\n' ' ' </tmp/fbx_list.txt)" | tar xzf - -C /tmp/fbx
BL=/Applications/Blender.app/Contents/MacOS/Blender     # 2. FBX -> per-clip skeleton-only GLB
"$BL" --background --python pipeline/clips_export.py -- --list pipeline/clips_r41.json --fbx /tmp/fbx --out /tmp/clip_glb
python3 pipeline/clips_pack.py --stage /tmp/clip_glb \  # 3. merge to <=6 groups + manifest
  --out web/models/clips --manifest web/assets/motion_manifest.json
python3 pipeline/clips_verify.py --stage /tmp/clip_glb  # 4. gate: 0 errors = green

No retarget is run and none is wanted: the Mixamo bank and PROCITY's peds are the same mixamorig skeleton (rigs.js _canon folds mixamorig1/4/…). A mixamorig→mixamorig pass through MIRPAMO would add error and bake the ped mesh into the clip file, destroying the zero-draw property. MIRPAMO (~/Documents/MIRPAMO, make smoke = its ground truth, green on Blender 5.1.2) is for foreign skeletons (CMU/BVH) and for rigging raw meshes. Blender-5 gotcha avoided by construction: FBX goes in, glTF comes out — nothing is ever exported through FBX.

Packer flags (each measured in AUDIT.md R41): --tracks all restores translation+scale (the engine's _rotOnly discards them, so they ship off by default) · --no-quantize keeps float32 rotations (int16-normalized is ~2× smaller at a worst measured error of 0.0034°) · --drop-fingers removes the 40 finger channels = 49.7% of the payload, taking the set to 1.71 MB. Accessor dedup is lossless and always on (saved 1 341 724 B).

⚠️ Licence zoning, ruling 1 (closed R41): MIRPAMO's library/downloads/ also held 3,077 CC BY-NC (never-shippable) Bandai-Namco BVH, unzoned and unmarked. They now live at library/downloads/red/bandai_namco_1_NC-research/ with the _NC-research suffix on every file (so mirpamo batch's <rig>@<clip>.glb naming propagates the marker), and MIRPAMO's library/manifest.json licence label is reconciled to ultra's MOTIONLIB/SOURCES.md. R41 ships one licence regime: Mixamo. CMU/100STYLE/ACCAD are CC-BY — they wait on 41.5's credits surface.

11. Palette — palette.py + isolate.py + montage.py (round-42 §42.1/§42.2a: DONE, free, on-device)

The R41 fittings shipped geometrically correct and visually unfinished. They were not untextured — they carry COLOR_0 and three.js renders it — they were bleached (15 of 37 pub props below 0.05 saturation). The fix is a palette imposed on colour that is already there, which costs zero draws: the COLOR_0 accessor is rewritten in place, so primitive count, triangle count and every POSITION/NORMAL/index byte are untouched. Adding baseColour maps instead would have re-imported the cost merge_prims.py bought back in R41 (14,140 → 117 interior draws).

python3 pipeline/palette.py --report web/assets/models/procity_fit_jukebox_01.glb   # what colour IS it?
python3 pipeline/palette.py --spec pipeline/r42_palette.json \
        --src pipeline/.props_orig/r42 --out pipeline/_normalized          # repaint the named set
python3 pipeline/strip_transmission.py --dry-run pipeline/_normalized/procity_fit_*.glb
python3 pipeline/check_contract.py --list pipeline/_normalized/procity_fit_deck_1200_rigged_01.glb
BL=/Applications/Blender.app/Contents/MacOS/Blender
"$BL" --background --python pipeline/thumbs.py -- web/assets/thumbs pipeline/_normalized/<set>.glb
GOD3_DEPOT=... GOD3_PW=... python3 pipeline/publish.py --files pipeline/_r42_publish_list.txt

⚠️ palette.py is NOT idempotent. ramp and tint normalise against the source's own luminance spread, so a second pass re-stretches an already-stretched file. Every output is stamped asset.extras.procity_palette; a stamped input is skipped unless --force. Point --src at the pristine copies in pipeline/.props_orig/r42/ (git-ignored) when re-running. Routes, per-asset stops/rules and the reason for each are in pipeline/r42_palette.json — every asset by name.

Proof tools: isolate.py renders ONE staged scene from a perspective camera at eye height on a bench of a stated colour (the only rig that can answer "does this read from where the player stands" — it is what diagnosed the DJ booth); montage.py builds labelled before/after grids from finished PNGs; contact_sheet.py --pitch M spaces tiles when the set mixes a 2.68 m umbrella with a 0.2 m ashtray.