modelbeast/AGENTS.md

# AGENTS.md — Using the M3 Ultra asset factory (MODELBEAST)

**Audience: any agent (or human) that needs 3D models, gaussian splats, camera-pose
datasets, generated/cleaned images, or format conversions.** This machine — an
M3 Ultra Mac Studio, 256GB unified memory — runs MODELBEAST, a job server that
wraps local Apple-Silicon ML tools and cloud APIs behind one HTTP API. You send
files and job requests; it returns finished assets.

**Fleet-aware?** This M3 is the primary of a 3-Mac Tailscale pool (M3 256GB + M1 128GB + M4 24GB). Read [CLUSTER.md](CLUSTER.md) for the local-first policy, node roles, and how the central queue distributes gpu jobs across machines.

## 1. Reaching it

- **Base URL (tailnet):** `http://100.89.131.57:8777` (M3 Ultra, primary) — from the machine itself, `http://localhost:8777`. Public (once the VPS is up): `https://modelbeast.digalot.fyi`.
- **Second node (M1 Ultra):** `http://100.91.239.7:8777` — an independent MODELBEAST instance for **local** operators (own login/library/token). ~1.5-1.8× slower than the M3 but doubles local throughput; run FLUX/SF3D/Brush/RMBG here to keep the M3 free. (TRELLIS.2 unverified on M1 — bf16.) A unified one-queue worker pool across both is the planned next step.
- **AUTH REQUIRED.** Every `/api/*` call needs a session cookie (browser login) or a bearer token (agents/CLI). Only `/api/health` and the static UI are public. Get a token from the owner: in the UI ⚙ Settings → Users → **token**, or `scripts/users.py`. Then `export MB_TOKEN=mbt_...`.
- **Guests are LOCAL-ONLY** by hard rule: a guest account can run only local operators (no fal/OpenRouter — those are owner-only and blocked server-side), sees only its own assets/jobs, and has a per-user concurrent-job cap. If you're acting as a guest, the cloud operators simply won't appear and will 403 if forced.
- Web UI at the same URL (humans log in). Agents use the REST API or the `mb` CLI with `MB_TOKEN`.
- If the server is down (connection refused): on the M3 run `~/Documents/MODELBEAST/scripts/serve.sh` (first run prints a one-time owner password to `data/server.log`).

## 2. The `mb` CLI (recommended)

One stdlib-only python3 file at repo root — copy it anywhere (`scp m3ultra:~/Documents/MODELBEAST/mb .`) and point it at the box:

```bash
export MB_HOST=http://100.89.131.57:8777
export MB_TOKEN=mbt_...      # from the owner (⚙ Settings → Users → token). Required — every call is authed.

./mb ops -v                 # operator catalog + every parameter (LIVE truth — trust this over any doc)
./mb upload photo.jpg       # -> prints asset id
./mb run fal_trellis2 --asset <ID> -p resolution=1024 --wait --download out/
./mb run flux_local -p prompt="a brass astrolabe" -p seed=7 --follow
./mb jobs                   # recent jobs      ./mb log <JOB> -f   # stream a log
./mb assets --kind model    # browse assets    ./mb get <ASSET> -o out/
./mb retry <JOB>  /  ./mb cancel <JOB>  /  ./mb rm-job <JOB>
```

- `mb run --file local.png ...` uploads and runs in one step.
- `--wait` blocks and prints output asset ids; `--download DIR` also fetches them; `--follow` streams the live log. Exit codes: 0 done, 2 error, 3 cancelled.
- `-p k=v` is type-coerced against the operator's schema; `--params-json '{...}'` for anything complex.

### Raw REST (if you can't run python)
```
GET  /api/operators                    # catalog incl. params_schema (JSON Schema)
POST /api/assets                       # multipart file upload -> asset
GET  /api/assets                       # list (outputs have parent_job=<job id>)
GET  /api/assets/{id}/file[?member=f]  # download (folder assets: JSON listing, then ?member=)
POST /api/jobs                         # {"operator": id, "asset_id": id|null, "params": {...}}
GET  /api/jobs/{id}                    # status: queued|running|done|error|cancelled + log
POST /api/jobs/{id}/cancel | /retry    # control
```
Poll `GET /api/jobs/{id}` every ~2s. When `done`, your outputs are the assets whose `parent_job` equals the job id.

## 3. What it can make (operator catalog — run `mb ops -v` for live params)

| Goal | Operator(s) | Notes |
|---|---|---|
| **image → 3D mesh (GLB)** | `fal_trellis2` (SOTA open, PBR, $0.05/unit — verified from real billing 2026-07) · `fal_hunyuan3d_v21` (~90s, PBR) · `fal_rodin` (hero-grade, $0.40+) · `fal_trellis` ($0.02 draft) · local: `trellis_mac`, `hunyuan3d_mlx` (native MLX, ~4.3 min on M3 Ultra, PBR, **no HF login** — weights public), `sf3d` | **Best practice: run `fal_bg_remove` first** and feed the cutout — biggest quality lever. `trellis_mac`/`sf3d` need the owner's HF token; `hunyuan3d_mlx` does not; fal ops need FAL_KEY (check `mb ops` for `[needs ...]` flags) |
| **video → gaussian splat** | `ffmpeg_frames` → `colmap_poses` → `brush_train` | 100% local/free. Orbit video, 2–3 fps sampling, 150–300 frames. 30k steps = quality, 1500 = fast preview |
| **video → camera poses** | `ffmpeg_frames` → `colmap_poses` | Outputs a COLMAP dataset folder (cameras/images/points3D) |
| **prompt → image** | `flux_local` (on-device, free, seed-reproducible; `flux2-klein-4b` works with zero keys) · `openrouter_image` (nano-banana family, ~$0.04–0.24/img) · `fal_text_image` (Ideogram, readable text) | A/B: same prompt through several, then judge |
| **image cleanup** | `fal_bg_remove` (cutout) · `fal_upscale` (faithful) · `fal_image_edit` (prompt edits: "remove the sticker") | All sub-cent to low-cent |
| **3D format conversion** | `blender_convert` | GLB/GLTF/OBJ/FBX/STL/PLY/USD/BLEND in → glb/fbx/obj/usd/usdz/stl/ply/blend out |
| **media inspection** | `ffprobe` | codec/resolution/fps/duration JSON |

## 4. Recipes

**Photo → clean game-ready mesh (the standard chain):**
```bash
CUT=$(./mb run fal_bg_remove --file product.jpg --wait | awk '/image/{print $1}' | tail -1)
./mb run fal_trellis2 --asset $CUT -p resolution=1024 --wait --download out/
```

**Orbit video → splat:**
```bash
VID=$(./mb upload orbit.mp4 | awk '{print $1}')
FRAMES=$(./mb run ffmpeg_frames --asset $VID -p fps=2.5 -p max_frames=250 --wait | awk '/frames/{print $1}' | tail -1)
DS=$(./mb run colmap_poses --asset $FRAMES --wait | awk '/colmap_dataset/{print $1}' | tail -1)
./mb run brush_train --asset $DS -p total_steps=30000 --wait --download out/
```

**Local vs cloud image A/B (fixed prompt):**
```bash
P="isometric low-poly tavern, warm lantern light"
./mb run flux_local -p prompt="$P" -p model=flux2-klein-4b -p seed=7 --wait
./mb run openrouter_image -p prompt="$P" -p model=google/gemini-2.5-flash-image --wait
# then compare in the web UI (⊞ Compare) or download both
```

**Drop-folder ingest (no API needed):** copy files into `~/Documents/MODELBEAST/data/inbox/` on the M3 — they auto-register as assets within ~6s (size-stable check). Useful for rsync/UE/Blender exports.

## 5. Job semantics you must respect

- **Concurrency lanes:** `gpu` jobs run ONE at a time (Metal contention — trellis_mac, sf3d, brush_train, flux_local); `cpu` 3 at once; `net` (cloud APIs) 6 at once. Queued jobs wait their turn — don't spam retries because something is `queued`.
- **First-run weight downloads** on local ML operators can take 10–30+ min (klein ~15GB, schnell ~24GB, TRELLIS ~15GB). The job log shows download progress; be patient, don't cancel.
- Logs stream into the job record (`mb log ID -f`). Secrets are redacted server-side.
- A job that ends `error` has the reason in `error` + last log lines. `mb retry ID` re-runs it with identical inputs.
- Server restarts re-queue anything that was running. Jobs are cheap to re-run — assets are the durable artifacts.

## 6. Keys, gating, and cost

- Gated operators are visible but refuse to run with a clear error until the owner adds the key in ⚙ Settings (`fal_key`, `openrouter_key`, `hf_token`). Check gating live via `mb ops` (`[needs ...]`).
- REAL costs (verified from the owner's fal billing, 2026-07): trellis2 $0.05/unit · trellis/multi $0.02 · hunyuan3d/v2 $0.16/gen · hunyuan3d/v2/multi-view $0.017/gen (the volume bargain) · hunyuan3d-v21 $0.30/gen · triposr $0.07 · birefnet ~$0.001 · rodin $0.40+ · nano-banana ~$0.039 · nano-banana-pro ~$0.134 · everything local = $0. FAL_KEY + openrouter_key are SET in settings (2026-07-13).
- **Never ask for or handle the raw keys.** They live in the server's settings vault.

## 7. Etiquette & limits (you are a guest on someone's workstation)

- Don't delete assets/jobs you didn't create.
- Big batches: stagger `gpu` work; the owner may be using the machine.
- Disk: weights + assets are large; if you generate hundreds of assets, download what you need and `mb rm-asset` your intermediates.
- The box also hosts Ollama (`http://100.89.131.57:11434`, qwen3:235b/32b/8b) if you need local LLM inference — separate service, same etiquette.
- Benchmarks so far: `BENCHMARKS.md`. Architecture and roadmap: `README.md`, `HANDOFF.md`, `PLAN.md`.

## 8. Ops runbook (on the M3)

```bash
cd ~/Documents/MODELBEAST
./scripts/serve.sh              # start/restart headless (log: data/server.log)
./tests/smoke.sh                # 12-check regression suite (~30s, isolated data dir)
./mb jobs                       # what's running
git pull && cd web && npm run build && cd .. && ./scripts/serve.sh   # deploy an update
```
Repo: `ssh://git@100.71.119.27:222/monster/modelbeast.git`. Heavy tools reinstall via `scripts/install_*.sh` (vendor/, venvs/, bin/ are not in git).