modelbeast/AGENTS.md

AGENTS.md — Using the M3 Ultra asset factory (MODELBEAST)

Audience: any agent (or human) that needs 3D models, gaussian splats, camera-pose datasets, generated/cleaned images, or format conversions. This machine — an M3 Ultra Mac Studio, 256GB unified memory — runs MODELBEAST, a job server that wraps local Apple-Silicon ML tools and cloud APIs behind one HTTP API. You send files and job requests; it returns finished assets.

1. Reaching it

• Base URL (tailnet): http://100.89.131.57:8777 — from the machine itself, http://localhost:8777.
• No auth. The tailnet is the security boundary. Do not expose this port publicly.
• Web UI at the same URL (humans). Agents use the REST API or the mb CLI.
• If the server is down (connection refused): on the M3 run ~/Documents/MODELBEAST/scripts/serve.sh (or ask the owner to run scripts/install_launchagent.sh once for boot persistence).

2. The mb CLI (recommended)

One stdlib-only python3 file at repo root — copy it anywhere (scp m3ultra:~/Documents/MODELBEAST/mb .) and point it at the box:

export MB_HOST=http://100.89.131.57:8777

./mb ops -v                 # operator catalog + every parameter (LIVE truth — trust this over any doc)
./mb upload photo.jpg       # -> prints asset id
./mb run fal_trellis2 --asset <ID> -p resolution=1024 --wait --download out/
./mb run flux_local -p prompt="a brass astrolabe" -p seed=7 --follow
./mb jobs                   # recent jobs      ./mb log <JOB> -f   # stream a log
./mb assets --kind model    # browse assets    ./mb get <ASSET> -o out/
./mb retry <JOB>  /  ./mb cancel <JOB>  /  ./mb rm-job <JOB>

• mb run --file local.png ... uploads and runs in one step.
• --wait blocks and prints output asset ids; --download DIR also fetches them; --follow streams the live log. Exit codes: 0 done, 2 error, 3 cancelled.
• -p k=v is type-coerced against the operator's schema; --params-json '{...}' for anything complex.

Raw REST (if you can't run python)

GET  /api/operators                    # catalog incl. params_schema (JSON Schema)
POST /api/assets                       # multipart file upload -> asset
GET  /api/assets                       # list (outputs have parent_job=<job id>)
GET  /api/assets/{id}/file[?member=f]  # download (folder assets: JSON listing, then ?member=)
POST /api/jobs                         # {"operator": id, "asset_id": id|null, "params": {...}}
GET  /api/jobs/{id}                    # status: queued|running|done|error|cancelled + log
POST /api/jobs/{id}/cancel | /retry    # control

Poll GET /api/jobs/{id} every ~2s. When done, your outputs are the assets whose parent_job equals the job id.

3. What it can make (operator catalog — run mb ops -v for live params)

Goal	Operator(s)	Notes
image → 3D mesh (GLB)	fal_trellis2 (SOTA open, PBR, $0.25–0.35) · fal_hunyuan3d_v21 (~90s, PBR) · fal_rodin (hero-grade, $0.40+) · fal_trellis ($0.02 draft) · local: trellis_mac, sf3d	Best practice: run fal_bg_remove first and feed the cutout — biggest quality lever. Local ops need the owner's HF token; fal ops need FAL_KEY (check mb ops for [needs ...] flags)
video → gaussian splat	ffmpeg_frames → colmap_poses → brush_train	100% local/free. Orbit video, 2–3 fps sampling, 150–300 frames. 30k steps = quality, 1500 = fast preview
video → camera poses	ffmpeg_frames → colmap_poses	Outputs a COLMAP dataset folder (cameras/images/points3D)
prompt → image	flux_local (on-device, free, seed-reproducible; flux2-klein-4b works with zero keys) · openrouter_image (nano-banana family, ~$0.04–0.24/img) · fal_text_image (Ideogram, readable text)	A/B: same prompt through several, then judge
image cleanup	fal_bg_remove (cutout) · fal_upscale (faithful) · fal_image_edit (prompt edits: "remove the sticker")	All sub-cent to low-cent
3D format conversion	blender_convert	GLB/GLTF/OBJ/FBX/STL/PLY/USD/BLEND in → glb/fbx/obj/usd/usdz/stl/ply/blend out
media inspection	ffprobe	codec/resolution/fps/duration JSON

4. Recipes

Photo → clean game-ready mesh (the standard chain):

CUT=$(./mb run fal_bg_remove --file product.jpg --wait | awk '/image/{print $1}' | tail -1)
./mb run fal_trellis2 --asset $CUT -p resolution=1024 --wait --download out/

Orbit video → splat:

VID=$(./mb upload orbit.mp4 | awk '{print $1}')
FRAMES=$(./mb run ffmpeg_frames --asset $VID -p fps=2.5 -p max_frames=250 --wait | awk '/frames/{print $1}' | tail -1)
DS=$(./mb run colmap_poses --asset $FRAMES --wait | awk '/colmap_dataset/{print $1}' | tail -1)
./mb run brush_train --asset $DS -p total_steps=30000 --wait --download out/

Local vs cloud image A/B (fixed prompt):

P="isometric low-poly tavern, warm lantern light"
./mb run flux_local -p prompt="$P" -p model=flux2-klein-4b -p seed=7 --wait
./mb run openrouter_image -p prompt="$P" -p model=google/gemini-2.5-flash-image --wait
# then compare in the web UI (⊞ Compare) or download both

Drop-folder ingest (no API needed): copy files into ~/Documents/MODELBEAST/data/inbox/ on the M3 — they auto-register as assets within ~6s (size-stable check). Useful for rsync/UE/Blender exports.

5. Job semantics you must respect

• Concurrency lanes: gpu jobs run ONE at a time (Metal contention — trellis_mac, sf3d, brush_train, flux_local); cpu 3 at once; net (cloud APIs) 6 at once. Queued jobs wait their turn — don't spam retries because something is queued.
• First-run weight downloads on local ML operators can take 10–30+ min (klein ~15GB, schnell ~24GB, TRELLIS ~15GB). The job log shows download progress; be patient, don't cancel.
• Logs stream into the job record (mb log ID -f). Secrets are redacted server-side.
• A job that ends error has the reason in error + last log lines. mb retry ID re-runs it with identical inputs.
• Server restarts re-queue anything that was running. Jobs are cheap to re-run — assets are the durable artifacts.

6. Keys, gating, and cost

• Gated operators are visible but refuse to run with a clear error until the owner adds the key in ⚙ Settings (fal_key, openrouter_key, hf_token). Check gating live via mb ops ([needs ...]).
• Rough costs when unlocked: fal_trellis $0.02 · hunyuan v21 ~$0.05–0.10 · trellis2 $0.25–0.35 · rodin $0.40+ · bg_remove/upscale sub-cent · nano-banana ~$0.039 · nano-banana-pro ~$0.134 · everything local = $0.
• Never ask for or handle the raw keys. They live in the server's settings vault.

7. Etiquette & limits (you are a guest on someone's workstation)

• Don't delete assets/jobs you didn't create.
• Big batches: stagger gpu work; the owner may be using the machine.
• Disk: weights + assets are large; if you generate hundreds of assets, download what you need and mb rm-asset your intermediates.
• The box also hosts Ollama (http://100.89.131.57:11434, qwen3:235b/32b/8b) if you need local LLM inference — separate service, same etiquette.
• Benchmarks so far: BENCHMARKS.md. Architecture and roadmap: README.md, HANDOFF.md, PLAN.md.

8. Ops runbook (on the M3)

cd ~/Documents/MODELBEAST
./scripts/serve.sh              # start/restart headless (log: data/server.log)
./tests/smoke.sh                # 12-check regression suite (~30s, isolated data dir)
./mb jobs                       # what's running
git pull && cd web && npm run build && cd .. && ./scripts/serve.sh   # deploy an update

Repo: ssh://git@100.71.119.27:222/monster/modelbeast.git. Heavy tools reinstall via scripts/install_*.sh (vendor/, venvs/, bin/ are not in git).