573afa7b7a
- server/operators/hunyuan3d_mlx: native-MLX Hunyuan3D 2.1, both stages, PBR. Verified end-to-end on M3 Ultra: 260s total, 40k-face GLB w/ 2048² PBR. Weights public (no HF login). Runs on M1 (MLX-native). - scripts/install_hunyuan3d_mlx.sh: uv py3.11 venv + MLX-path deps. - CLUSTER.md: local-first policy + M3/M1/M4 fleet roles + central-queue howto. - BENCHMARKS/README/AGENTS: hunyuan3d_mlx rows + trellis head-to-head.
123 lines
9.3 KiB
Markdown
123 lines
9.3 KiB
Markdown
# AGENTS.md — Using the M3 Ultra asset factory (MODELBEAST)
|
||
|
||
**Audience: any agent (or human) that needs 3D models, gaussian splats, camera-pose
|
||
datasets, generated/cleaned images, or format conversions.** This machine — an
|
||
M3 Ultra Mac Studio, 256GB unified memory — runs MODELBEAST, a job server that
|
||
wraps local Apple-Silicon ML tools and cloud APIs behind one HTTP API. You send
|
||
files and job requests; it returns finished assets.
|
||
|
||
**Fleet-aware?** This M3 is the primary of a 3-Mac Tailscale pool (M3 256GB + M1 128GB + M4 24GB). Read [CLUSTER.md](CLUSTER.md) for the local-first policy, node roles, and how the central queue distributes gpu jobs across machines.
|
||
|
||
## 1. Reaching it
|
||
|
||
- **Base URL (tailnet):** `http://100.89.131.57:8777` (M3 Ultra, primary) — from the machine itself, `http://localhost:8777`. Public (once the VPS is up): `https://modelbeast.digalot.fyi`.
|
||
- **Second node (M1 Ultra):** `http://100.91.239.7:8777` — an independent MODELBEAST instance for **local** operators (own login/library/token). ~1.5-1.8× slower than the M3 but doubles local throughput; run FLUX/SF3D/Brush/RMBG here to keep the M3 free. (TRELLIS.2 unverified on M1 — bf16.) A unified one-queue worker pool across both is the planned next step.
|
||
- **AUTH REQUIRED.** Every `/api/*` call needs a session cookie (browser login) or a bearer token (agents/CLI). Only `/api/health` and the static UI are public. Get a token from the owner: in the UI ⚙ Settings → Users → **token**, or `scripts/users.py`. Then `export MB_TOKEN=mbt_...`.
|
||
- **Guests are LOCAL-ONLY** by hard rule: a guest account can run only local operators (no fal/OpenRouter — those are owner-only and blocked server-side), sees only its own assets/jobs, and has a per-user concurrent-job cap. If you're acting as a guest, the cloud operators simply won't appear and will 403 if forced.
|
||
- Web UI at the same URL (humans log in). Agents use the REST API or the `mb` CLI with `MB_TOKEN`.
|
||
- If the server is down (connection refused): on the M3 run `~/Documents/MODELBEAST/scripts/serve.sh` (first run prints a one-time owner password to `data/server.log`).
|
||
|
||
## 2. The `mb` CLI (recommended)
|
||
|
||
One stdlib-only python3 file at repo root — copy it anywhere (`scp m3ultra:~/Documents/MODELBEAST/mb .`) and point it at the box:
|
||
|
||
```bash
|
||
export MB_HOST=http://100.89.131.57:8777
|
||
export MB_TOKEN=mbt_... # from the owner (⚙ Settings → Users → token). Required — every call is authed.
|
||
|
||
./mb ops -v # operator catalog + every parameter (LIVE truth — trust this over any doc)
|
||
./mb upload photo.jpg # -> prints asset id
|
||
./mb run fal_trellis2 --asset <ID> -p resolution=1024 --wait --download out/
|
||
./mb run flux_local -p prompt="a brass astrolabe" -p seed=7 --follow
|
||
./mb jobs # recent jobs ./mb log <JOB> -f # stream a log
|
||
./mb assets --kind model # browse assets ./mb get <ASSET> -o out/
|
||
./mb retry <JOB> / ./mb cancel <JOB> / ./mb rm-job <JOB>
|
||
```
|
||
|
||
- `mb run --file local.png ...` uploads and runs in one step.
|
||
- `--wait` blocks and prints output asset ids; `--download DIR` also fetches them; `--follow` streams the live log. Exit codes: 0 done, 2 error, 3 cancelled.
|
||
- `-p k=v` is type-coerced against the operator's schema; `--params-json '{...}'` for anything complex.
|
||
|
||
### Raw REST (if you can't run python)
|
||
```
|
||
GET /api/operators # catalog incl. params_schema (JSON Schema)
|
||
POST /api/assets # multipart file upload -> asset
|
||
GET /api/assets # list (outputs have parent_job=<job id>)
|
||
GET /api/assets/{id}/file[?member=f] # download (folder assets: JSON listing, then ?member=)
|
||
POST /api/jobs # {"operator": id, "asset_id": id|null, "params": {...}}
|
||
GET /api/jobs/{id} # status: queued|running|done|error|cancelled + log
|
||
POST /api/jobs/{id}/cancel | /retry # control
|
||
```
|
||
Poll `GET /api/jobs/{id}` every ~2s. When `done`, your outputs are the assets whose `parent_job` equals the job id.
|
||
|
||
## 3. What it can make (operator catalog — run `mb ops -v` for live params)
|
||
|
||
| Goal | Operator(s) | Notes |
|
||
|---|---|---|
|
||
| **image → 3D mesh (GLB)** | `fal_trellis2` (SOTA open, PBR, $0.05/unit — verified from real billing 2026-07) · `fal_hunyuan3d_v21` (~90s, PBR) · `fal_rodin` (hero-grade, $0.40+) · `fal_trellis` ($0.02 draft) · local: `trellis_mac`, `hunyuan3d_mlx` (native MLX, ~4.3 min on M3 Ultra, PBR, **no HF login** — weights public), `sf3d` | **Best practice: run `fal_bg_remove` first** and feed the cutout — biggest quality lever. `trellis_mac`/`sf3d` need the owner's HF token; `hunyuan3d_mlx` does not; fal ops need FAL_KEY (check `mb ops` for `[needs ...]` flags) |
|
||
| **video → gaussian splat** | `ffmpeg_frames` → `colmap_poses` → `brush_train` | 100% local/free. Orbit video, 2–3 fps sampling, 150–300 frames. 30k steps = quality, 1500 = fast preview |
|
||
| **video → camera poses** | `ffmpeg_frames` → `colmap_poses` | Outputs a COLMAP dataset folder (cameras/images/points3D) |
|
||
| **prompt → image** | `flux_local` (on-device, free, seed-reproducible; `flux2-klein-4b` works with zero keys) · `openrouter_image` (nano-banana family, ~$0.04–0.24/img) · `fal_text_image` (Ideogram, readable text) | A/B: same prompt through several, then judge |
|
||
| **image cleanup** | `fal_bg_remove` (cutout) · `fal_upscale` (faithful) · `fal_image_edit` (prompt edits: "remove the sticker") | All sub-cent to low-cent |
|
||
| **3D format conversion** | `blender_convert` | GLB/GLTF/OBJ/FBX/STL/PLY/USD/BLEND in → glb/fbx/obj/usd/usdz/stl/ply/blend out |
|
||
| **media inspection** | `ffprobe` | codec/resolution/fps/duration JSON |
|
||
|
||
## 4. Recipes
|
||
|
||
**Photo → clean game-ready mesh (the standard chain):**
|
||
```bash
|
||
CUT=$(./mb run fal_bg_remove --file product.jpg --wait | awk '/image/{print $1}' | tail -1)
|
||
./mb run fal_trellis2 --asset $CUT -p resolution=1024 --wait --download out/
|
||
```
|
||
|
||
**Orbit video → splat:**
|
||
```bash
|
||
VID=$(./mb upload orbit.mp4 | awk '{print $1}')
|
||
FRAMES=$(./mb run ffmpeg_frames --asset $VID -p fps=2.5 -p max_frames=250 --wait | awk '/frames/{print $1}' | tail -1)
|
||
DS=$(./mb run colmap_poses --asset $FRAMES --wait | awk '/colmap_dataset/{print $1}' | tail -1)
|
||
./mb run brush_train --asset $DS -p total_steps=30000 --wait --download out/
|
||
```
|
||
|
||
**Local vs cloud image A/B (fixed prompt):**
|
||
```bash
|
||
P="isometric low-poly tavern, warm lantern light"
|
||
./mb run flux_local -p prompt="$P" -p model=flux2-klein-4b -p seed=7 --wait
|
||
./mb run openrouter_image -p prompt="$P" -p model=google/gemini-2.5-flash-image --wait
|
||
# then compare in the web UI (⊞ Compare) or download both
|
||
```
|
||
|
||
**Drop-folder ingest (no API needed):** copy files into `~/Documents/MODELBEAST/data/inbox/` on the M3 — they auto-register as assets within ~6s (size-stable check). Useful for rsync/UE/Blender exports.
|
||
|
||
## 5. Job semantics you must respect
|
||
|
||
- **Concurrency lanes:** `gpu` jobs run ONE at a time (Metal contention — trellis_mac, sf3d, brush_train, flux_local); `cpu` 3 at once; `net` (cloud APIs) 6 at once. Queued jobs wait their turn — don't spam retries because something is `queued`.
|
||
- **First-run weight downloads** on local ML operators can take 10–30+ min (klein ~15GB, schnell ~24GB, TRELLIS ~15GB). The job log shows download progress; be patient, don't cancel.
|
||
- Logs stream into the job record (`mb log ID -f`). Secrets are redacted server-side.
|
||
- A job that ends `error` has the reason in `error` + last log lines. `mb retry ID` re-runs it with identical inputs.
|
||
- Server restarts re-queue anything that was running. Jobs are cheap to re-run — assets are the durable artifacts.
|
||
|
||
## 6. Keys, gating, and cost
|
||
|
||
- Gated operators are visible but refuse to run with a clear error until the owner adds the key in ⚙ Settings (`fal_key`, `openrouter_key`, `hf_token`). Check gating live via `mb ops` (`[needs ...]`).
|
||
- REAL costs (verified from the owner's fal billing, 2026-07): trellis2 $0.05/unit · trellis/multi $0.02 · hunyuan3d/v2 $0.16/gen · hunyuan3d/v2/multi-view $0.017/gen (the volume bargain) · hunyuan3d-v21 $0.30/gen · triposr $0.07 · birefnet ~$0.001 · rodin $0.40+ · nano-banana ~$0.039 · nano-banana-pro ~$0.134 · everything local = $0. FAL_KEY + openrouter_key are SET in settings (2026-07-13).
|
||
- **Never ask for or handle the raw keys.** They live in the server's settings vault.
|
||
|
||
## 7. Etiquette & limits (you are a guest on someone's workstation)
|
||
|
||
- Don't delete assets/jobs you didn't create.
|
||
- Big batches: stagger `gpu` work; the owner may be using the machine.
|
||
- Disk: weights + assets are large; if you generate hundreds of assets, download what you need and `mb rm-asset` your intermediates.
|
||
- The box also hosts Ollama (`http://100.89.131.57:11434`, qwen3:235b/32b/8b) if you need local LLM inference — separate service, same etiquette.
|
||
- Benchmarks so far: `BENCHMARKS.md`. Architecture and roadmap: `README.md`, `HANDOFF.md`, `PLAN.md`.
|
||
|
||
## 8. Ops runbook (on the M3)
|
||
|
||
```bash
|
||
cd ~/Documents/MODELBEAST
|
||
./scripts/serve.sh # start/restart headless (log: data/server.log)
|
||
./tests/smoke.sh # 12-check regression suite (~30s, isolated data dir)
|
||
./mb jobs # what's running
|
||
git pull && cd web && npm run build && cd .. && ./scripts/serve.sh # deploy an update
|
||
```
|
||
Repo: `ssh://git@100.71.119.27:222/monster/modelbeast.git`. Heavy tools reinstall via `scripts/install_*.sh` (vendor/, venvs/, bin/ are not in git).
|