discgod/README.md

# discgod

A lean, all-in-one Discogs **seller inventory tracker**. Paste a Discogs API
token, pick the sellers you care about (Aussie or anyone), and discgod pulls
their **full inventories from the official Discogs API** on a schedule into the
`discogs_full` Postgres DB — building seller profiles and tracking price
changes and sales over time. No buttons, no DOM scraping.

It's the token-driven successor to **pliceclogs / marketwatcher**: same database,
same proven `mp_*` schema and upsert/diff/sold-detection engine, but the data
now comes from the API (robust, no fragile selectors, no Cloudflare) instead of
scraping rendered pages.

```
┌─────────────────────────┐    Discogs API (your token, 60/min)
│  extension (background) │ ───────────────────────────────────►  api.discogs.com
│  • holds token          │ ◄───────────────────────────────────  /users/{u}/inventory
│  • schedules + paginates│
│  • rate-limits + queue  │    raw listing pages (HTTP POST)
└───────────┬─────────────┘ ─────────────►  ┌────────────────────┐
            │                                │  discgod_server.py │ ──►  discogs_full
   popup = control panel                     │  (maps + upserts)  │      (mp_* tables)
                                             └────────────────────┘
            optional, for 24/7 even with the browser closed:
            discgod_crawl.py  ──(same engine, server-side)──►   discogs_full
```

## Layout

```
discgod/
  extension/            Chrome MV3 extension (load unpacked)
    manifest.json
    background.js        engine: token, scheduler, API pagination, rate-limit, retry queue
    popup.html/.css/.js  control panel: token, watchlist, scrape-now, live stats
    content.js           "▶ Track in discgod" button on Discogs seller pages
    icons/
  server/
    discgod_server.py    STANDALONE single-file ingest server (port 5057) —
                         schema + API→mp_* mapping + upsert/diff/gone + watchlist
                         + HTTP, all in one. Coexists with the legacy :5002 server;
                         paste its handlers into pliceclogs_server.py to merge later.
    discgod_crawl.py     headless crawler for cron / 24-7 (imports the engine
                         from discgod_server; no browser needed)
    schema.sql           reference (server auto-applies it on startup)
    requirements.txt     just psycopg2
  seeds/
    aus_sellers.txt      top 100 Australian sellers, ready to seed the watchlist
  deploy.sh              backup + deploy + restart the server on the tailscale box
```

## Quick start

### 1. Server (writes Postgres)
The Ultra (`ultra.local` = tailscale `100.91.239.7`) hosts `discogs_full`
locally, so there's nothing to deploy — just run the standalone server *on the
Ultra itself*. It listens on **:5057** and coexists with the legacy plice server
on :5002 (separate process, separate port, same DB, untouched legacy endpoints):

```bash
cd server
nohup /opt/homebrew/bin/python3 discgod_server.py > discgod_server.log 2>&1 &
curl -s localhost:5057/discgod/health   # {"ok":true,"server":"discgod",...}
```
It creates the `mp_*` tables and `discgod_seller_watch` in `discogs_full` on
startup (additive — touches nothing existing). `deploy.sh` is only needed if you
ever run the server on a *different* box.

To keep it running across reboots, either add that `nohup` line to your
MONSTERPANEL startup alongside the plice server, or wrap it in a launchd plist.

### 2. Extension (the engine + control panel)
1. `chrome://extensions` → enable Developer mode → **Load unpacked** → pick `discgod/extension`.
2. Open the popup, paste your **Discogs API token**
   ([Settings → Developers](https://www.discogs.com/settings/developers) → *Generate token*),
   click **Test** (should say "signed in as …").
3. Set the **Ingest server URL** (default `http://100.91.239.7:5057`) and an
   interval, tick **Auto-crawl**, **Save**.
4. Add sellers — type a username (or paste a `/seller/NAME` URL) and **Track**,
   or hit the **▶ Track in discgod** button while on any Discogs seller page.
   Add an optional filter like `format=Vinyl` or `ships_from=Australia`.
5. **Scrape all now**, or wait for the schedule. Watch the live stats fill in.

That's the whole "add token and go" loop.

### 3. (Optional) 24/7 headless crawling on the box
The extension only runs while Chrome is open. For always-on tracking, run the
crawler on the box — it shares the same watchlist:

```bash
ssh johnking@100.91.239.7
echo YOUR_TOKEN > ~/.discgod_token && chmod 600 ~/.discgod_token
cd /Users/johnking/Documents/MONSTERPANEL/discgod

python3 discgod_crawl.py --seed /tmp/aus_sellers.txt   # seed the 100 Aussie sellers
python3 discgod_crawl.py                                # crawl everything once
python3 discgod_crawl.py Dennis-brisbane-qld           # just one seller
python3 discgod_crawl.py --loop 43200                   # crawl all, repeat every 12 h
```

Or cron it (daily 3am):
```
0 3 * * *  cd /Users/johnking/Documents/MONSTERPANEL/discgod && /opt/homebrew/bin/python3 discgod_crawl.py >> crawl.log 2>&1
```

## What gets captured

| Table | Contents |
|---|---|
| `mp_listing` | One row per listing (`item_id` PK, `release_id` joins to your XML `release` table). Condition, sleeve, notes, price/currency, shipping, converted total, have/want, ships-from, seller. `first_seen` / `last_seen` / `times_seen` / `status` (`active`/`gone`). |
| `mp_listing_event` | Append-only history: `listed`, `price_change` (old/new), `condition_change`, `gone` (presumed sold/delisted, with last price), `relisted`. |
| `mp_seller` | Each seller seen — id, username, rating, ships-from. |
| `mp_seller_snapshot` | Inventory-size time series (`source='api_inventory'`), from the API's authoritative pagination total. Deduped within 6 h per (seller, filter). |
| `mp_page_log` | Provenance: every captured inventory page + the item_ids it held. |
| `discgod_seller_watch` | The watchlist: who to crawl, optional per-seller filter, last-crawled status. |

**Sold detection:** every full crawl stamps each listing with a `crawl_id`. When
the crawl finishes, listings the crawl *didn't* see (`last_crawl_id` differs) are
flipped to `gone` with a `gone` event — your sold/delisted feed. Filtered crawls
(e.g. `format=Vinyl`) skip gone-marking unless they cover the seller's whole
inventory.

## Useful queries

```sql
-- A seller's inventory size over time
SELECT captured_at::date, max(total_listings)
FROM mp_seller_snapshot
WHERE username = 'Dennis-brisbane-qld' AND source = 'api_inventory'
GROUP BY 1 ORDER BY 1;

-- Presumed sales last 30 days, with full release info from the XML dump
SELECT e.recorded_at::date, e.seller_username, e.old_price, r.title, ra.artist_name
FROM mp_listing_event e
JOIN release r ON r.id = e.release_id
LEFT JOIN release_artist ra ON ra.release_id = r.id AND ra.position = 1
WHERE e.event = 'gone' AND e.recorded_at > now() - interval '30 days'
ORDER BY e.recorded_at DESC;

-- Price drops
SELECT e.recorded_at, e.seller_username, e.old_price, e.new_price, r.title
FROM mp_listing_event e JOIN release r ON r.id = e.release_id
WHERE e.event = 'price_change' AND e.new_price < e.old_price
ORDER BY e.recorded_at DESC LIMIT 50;
```

## Notes & limits

- **Rate limit:** the Discogs API allows ~60 authenticated requests/min. discgod
  paces itself (~55/min) and backs off on 429. A 26k-listing seller is ~270 API
  pages ≈ 5 min — fine periodically; the headless crawler is better for big
  bulk sweeps.
- **Public inventories only** — the API returns another user's listings that are
  "For Sale". That's every real seller.
- The extension's background worker can be suspended by Chrome on very long
  crawls; discgod persists the per-seller queue and a 1-minute keepalive so a
  sweep resumes where it left off. For unattended bulk work, prefer the headless
  crawler.
- Off-box server? Set `PGHOST`/`PGUSER`/`PGPASSWORD` (and `DISCGOD_DB`) before
  launching `discgod_server.py`. On the box it uses local peer auth, no creds.