Schedule the depot archive, and add a prune that cannot 404 a shipping game

SCHEDULE. games.monsterrobot.depot-archive installed on ultra, daily 05:30 — after meshgod at
05:00, matching the existing games.monsterrobot.* launchd convention, plist kept in the repo
beside the script. plutil-linted and bootstrapped; launchctl reports it registered.

PRUNE. John wants old GLBs off the live VPS while keeping everything backed up. Age alone cannot
decide that: two shipping games resolve depot assets BY FILENAME at runtime, so a three-year-old
mesh some game still names is load-bearing, not stale, and deleting it is a 404 in a released
build. So prune.py deletes only when all three hold — older than --days, named in NO consumer's
source (scanning js/html/json/ts across thriftgod, procity and 90sDJsim, including the JSON
manifests that also carry depot refs), and an identical-SIZE copy confirmed in the Drive archive.
Size mismatch counts as unbacked; it never deletes on a maybe. Dry-run by default, and apply
re-verifies every rule rather than trusting the earlier plan. Thumbnails and meta.json are left
alone so the depot keeps listing pruned entries, as the MeshGod archive does.

Measured today it deletes NOTHING, correctly: of 774 assets, 35 are game-referenced and 724 are
newer than a 180-day cutoff — every asset in the depot is under 30 days old. Reported as-is
rather than loosening the cutoff to manufacture a result.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
type-two 2026-07-24 22:51:35 +10:00
parent 03772cff9b
commit 9423cf8f64
3 changed files with 150 additions and 0 deletions

View File

@ -78,3 +78,27 @@ Verify freshness the same way as the other flows:
⚠️ Shares the whole-fleet risk noted in the backup-verify skill: the `gdrive:` remote uses
rclone's shared Google client_id, which Google retires during 2026. When it dies every Drive flow
breaks at once, this one included.
### Schedule
`games.monsterrobot.depot-archive` — launchd on ultra, daily **05:30** (after meshgod at 05:00),
logging to `~/depot-backups/launchd.log`. Plist kept here alongside the script.
### Pruning (`prune.py`)
Reclaims VPS disk **without** the 404 risk, because age alone is not a safe criterion — a
three-year-old asset a game still names is load-bearing, not stale. A GLB is deleted only when
**all three** hold:
1. older than `--days`
2. its name appears in **no** consumer's source (js/html/json/ts across thriftgod, procity, 90sDJsim)
3. an **identical-size copy is confirmed in the Drive archive**
Dry-run by default (`plan`); `apply` re-verifies every rule before deleting. Thumbnails and
`meta.json` are never touched, so the depot keeps listing pruned assets — same behaviour as the
MeshGod archive.
```bash
python3 ~/depot-backups/prune.py plan --days 180 # on ultra, where rclone lives
```
As of 2026-07-24 this deletes **nothing**: of 774 assets, 35 are game-referenced and 724 are
newer than the cutoff. Every asset in the depot is under 30 days old. The mechanism is armed for
when that changes.

View File

@ -0,0 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
<key>Label</key><string>games.monsterrobot.depot-archive</string>
<key>ProgramArguments</key>
<array><string>/bin/sh</string><string>/Users/johnking/depot-backups/archive.sh</string></array>
<key>EnvironmentVariables</key>
<dict><key>PATH</key><string>/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin</string></dict>
<key>StartCalendarInterval</key><dict><key>Hour</key><integer>5</integer><key>Minute</key><integer>30</integer></dict>
<key>StandardOutPath</key><string>/Users/johnking/depot-backups/launchd.log</string>
<key>StandardErrorPath</key><string>/Users/johnking/depot-backups/launchd.log</string>
<key>WorkingDirectory</key><string>/Users/johnking</string>
</dict></plist>

113
depot/prune.py Normal file
View File

@ -0,0 +1,113 @@
#!/usr/bin/env python3
"""Prune old depot GLBs from the live VPS — safely.
Age alone is NOT a safe criterion here. Two shipping games resolve depot assets BY FILENAME at
runtime (thriftgod web/index.html:1176, procity web/js/core/loaders.js), so deleting a mesh some
game still names is a 404 in a released build. A three-year-old asset that a game references is
not stale, it is load-bearing.
So a file is only ever deleted when ALL of these hold:
1. it is older than --days
2. its name appears in NO consumer's source (js/html/json/ts across every game repo)
3. an identical-size copy is confirmed present in the Drive archive
Dry-run by default. Thumbnails and meta.json are never touched, so the depot keeps listing the
asset matching how the MeshGod archive behaves after a prune.
prune.py plan [--days 180]
prune.py apply [--days 180] # actually deletes, after re-verifying every rule
"""
import json, os, re, subprocess, sys, time, urllib.parse, urllib.request
DEPOT = os.environ.get('WG_GOD3', 'http://100.94.195.115:8788')
DRIVE = os.environ.get('WG_DEPOT_DRIVE', 'gdrive:DB-BACKUP/3god-depot')
RCLONE = os.environ.get('WG_RCLONE', '/opt/homebrew/bin/rclone')
ULTRA = 'johnking@100.91.239.7'
# Every tree that can name a depot asset. Local paths are on JING5; ultra ones are fetched by ssh.
LOCAL_SOURCES = [os.path.expanduser('~/Documents/thriftgod/web'),
os.path.expanduser('~/Documents/PROCITY/web')]
ULTRA_SOURCES = ['Documents/90sDJsim/web', 'Documents/thriftgod/web']
GLB_RE = r'[A-Za-z0-9._-]+\.glb'
def referenced():
"""Every .glb filename mentioned anywhere in any consumer's source."""
names = set()
for d in LOCAL_SOURCES:
if not os.path.isdir(d):
continue
r = subprocess.run(['grep', '-rhoaE', GLB_RE, d], capture_output=True, text=True)
names |= {x.strip() for x in r.stdout.splitlines() if x.strip()}
for d in ULTRA_SOURCES:
r = subprocess.run(['ssh', '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=10', ULTRA,
f'grep -rhoaE "{GLB_RE}" ~/{d} 2>/dev/null'],
capture_output=True, text=True)
names |= {x.strip() for x in r.stdout.splitlines() if x.strip()}
return names
def depot_list():
with urllib.request.urlopen(DEPOT + '/api/list', timeout=60) as r:
return json.load(r).get('assets', [])
def drive_index():
"""{name: size} from the Drive archive — the proof a delete is recoverable."""
r = subprocess.run([RCLONE, 'lsjson', DRIVE], capture_output=True, text=True, timeout=600)
if r.returncode != 0:
raise SystemExit(f'cannot read the Drive archive ({DRIVE}): {r.stderr.strip()[:200]}\n'
'Refusing to plan a prune without a verified backup.')
return {e['Name']: e['Size'] for e in json.loads(r.stdout or '[]')}
def plan(days):
refs, assets, drive = referenced(), depot_list(), drive_index()
cutoff = time.time() - days * 86400
out = {'days': days, 'depot_total': len(assets), 'referenced': len(refs),
'drive_objects': len(drive), 'delete': [], 'kept': {}}
keep_ref = keep_young = keep_unbacked = 0
for a in assets:
f = a.get('file', '')
if not f.lower().endswith('.glb'):
continue
if f in refs:
keep_ref += 1; continue
if a.get('added', 0) > cutoff:
keep_young += 1; continue
if drive.get(f) != a.get('size'):
# present-but-different-size counts as unbacked; never delete on a maybe
keep_unbacked += 1; continue
out['delete'].append({'file': f, 'size': a.get('size'),
'age_days': int((time.time() - a.get('added', 0)) / 86400)})
out['kept'] = {'referenced_by_a_game': keep_ref, 'newer_than_cutoff': keep_young,
'NOT_in_drive_backup': keep_unbacked}
out['reclaim_bytes'] = sum(d['size'] for d in out['delete'])
return out
def apply(days):
p = plan(days)
ok, fail = [], []
for d in p['delete']:
try:
req = urllib.request.Request(
DEPOT + '/api/del?name=' + urllib.parse.quote(d['file']), method='POST')
with urllib.request.urlopen(req, timeout=60) as r:
r.read()
ok.append(d['file'])
except Exception as e:
fail.append({'file': d['file'], 'error': str(e)[:120]})
p['deleted'], p['failed'] = ok, fail
return p
if __name__ == '__main__':
cmd = sys.argv[1] if len(sys.argv) > 1 else 'plan'
days = int(sys.argv[sys.argv.index('--days') + 1]) if '--days' in sys.argv else 180
res = apply(days) if cmd == 'apply' else plan(days)
res['reclaim_gb'] = round(res.get('reclaim_bytes', 0) / 1073741824, 2)
sample = res['delete'][:8]
res['delete_count'] = len(res['delete'])
res['delete'] = sample if cmd != 'apply' else res['delete'][:8]
print(json.dumps(res, indent=1))