The gpu lane is now a pool of nodes -- this Mac plus any remote workers in
nodes.json (gitignored, primary-only). A gpu job acquires the first free node
that supports its operator; local runs as before, a remote node has its inputs
rsynced out, runs the operator's run.py over ssh (keys sourced from the node's
.env.remote, kept off the process table), then its outputs are rsynced back and
registered locally exactly as for a local job. One job per node (one Metal
device each).
- server/remote.py (new): node loading, cached health checks, prepare/cmd/collect
- runner: _acquire_gpu_node + a remote branch in _run_job. lane_of now treats
'gpu' as a real lane -- it was dropped from LANE_LIMITS (which holds only the
semaphore lanes cpu/net), so gpu jobs were silently falling back to the cpu
lane and the pool was never reached. A node is reserved before the health
await so two concurrent jobs can't grab the same one.
- sysinfo: gpu lane capacity = pool size, plus per-node status for the dashboard
Verified: all 7 gpu operators route to the pool; remote_cmd/node_supports and
nodes.json parsing correct; acquisition is race-free; a scratch-DB server boots
clean and logs the pool.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
