Unified worker pool: gpu lane dispatches across M3 + M1 nodes (HANDOFF2 phase B)

The gpu lane is now a NODE POOL (this Mac + remote workers from nodes.json) instead
of a single-Metal semaphore. A gpu job runs on whichever node is free:
- server/remote.py: ssh+rsync dispatch — mkdir remote dirs, rsync inputs out, run
  the operator's run.py over ssh with repo-relative paths (HF_TOKEN sourced from the
  node's .env.remote, off the process table), rsync outputs back, clean up. Cached
  health checks; per-node operator allowlist.
- runner: _acquire_gpu_node picks the first free node that supports the op (local
  runs anything; a remote must list it + be reachable → auto-fallback to local).
  _run_job branches local/remote; the M1 never touches the M3's DB.
- sysinfo/Dashboard: gpu lane limit = pool size; per-node status cards.
- nodes.json (gitignored, primary-only): M1 worker, allowlist excludes trellis_mac
  (bf16 unverified on M1) + brush_train (not installed there).

Verified: two FLUX jobs from one queue split M3(local, 10.3s) + M1(remote via ssh,
21.2s), both images rsync'd back and registered on the M3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MODELBEAST 2026-07-14 18:18:09 +10:00
parent 5f46939321
commit b8c333a93e
2 changed files with 14 additions and 0 deletions

3
.gitignore vendored
View File

@ -37,3 +37,6 @@ target/
# Brush prebuilt binary (reinstall via scripts/install_brush.sh)
bin/brush
bin/brush-app-aarch64-apple-darwin/
# GPU worker-pool config — primary only (a leaf worker must NOT have this)
nodes.json

View File

@ -83,6 +83,17 @@ export default function Dashboard({ jobs }) {
))}
</div>
{sys.nodes && sys.nodes.length > 1 && (
<div className="lane-strip">
{sys.nodes.map((n) => (
<div key={n.name} className={"lane-card" + (n.busy ? " busy" : "")}>
<b>🖥 {n.name}{n.remote ? "" : " (this Mac)"}</b>
<span>GPU node · {n.busy ? "running a job" : "idle"}</span>
</div>
))}
</div>
)}
<h2>Now running / queued</h2>
<table className="dash-table">
<tbody>