RECORDGOD/docs/RAREWAVES_INGEST_PLAN.md

Distro ingest (RareWaves first) + cost tracking + new/used — plan (2026-06-27)

Buy stock from distros (RareWaves discount sales, ~$8–15 capped shipping → margin in volume), scrape the order + wishlist account pages via the PRICEGOD extension, land cost price per copy in inventory, and use the wishlist to gauge scarcity (how many stores / Discogs sellers stock it). Prep Rocket / Inertia / Bertus next (they send invoices to ingest).

✅ Done (foundations, live on recordgod)

• inventory.cost_price numeric(10,2) — per-COPY buy price (costs vary per copy, so inventory-level not release-level).
• inventory.cost_source text — provenance, e.g. rarewaves #592619.
• inventory.condition_type text NOT NULL DEFAULT 'used' — the new/used binary split. Backfilled: 451 sealed items (notes ILIKE '%seal%') → new; everything else used. (Distro-bought stock defaults new.)

What the RareWaves dumps revealed (analysis)

RareWaves = Shopify. Source files on ultra ~/Documents/recordgod/rarw*.txt.

Orders (rarworder.txt)* — new Shopify customer-account React pages. Each line item:

<a href="/products/0603497816163-rumours-2025?variant=55398669779318">
   <img alt="Rumours">   …   quantity 4   …   $42.99/ea

→ barcode (EAN-13 = leading digits of the handle) + title (img alt) + year (in slug) + qty + unit cost ($X.XX/ea) + variant id + order # (Order #592619). CSS classes are obfuscated hashes — key off the /products/<barcode>- URL, the /ea price text, ARIA role="row", and quantity N. High confidence.

Wishlist (rarwwish.txt)* — rendered by the Klevu app into collapsible data-wishlist-row-id rows. No clean barcode in the dump (the 13-digit hits were CDN cache-bust hashes ?v=…, false positives). Harder — needs row→product mapping, likely via Klevu's data or by following each row's product link. Lower confidence; phase 2.

Build plan (sequenced)

1. Order scraper → cost into inventory (HIGH value, HIGH confidence — do first).

• PRICEGOD content script on rarewaves.com/account/orders/* (+ legacy /orders/*): walk each role="row", pull {barcode, title, year(slug), qty, unit_cost, variant_id} + the order number.
• New RecordGod endpoint POST /admin/intake/distro {source:'rarewaves', order_ref, items:[…]} → per item: resolve barcode → release_id (local disc_release_identifier type='Barcode', normalised: strip non-digits, try with/without leading 0; Discogs /database/search?barcode= fallback), enrich, stage with cost_price=unit_cost, cost_source='rarewaves #<order>', condition_type='new', one SKU per copy (qty → N rows).
• Reuses the existing _stage/_enrich + the staged review queue. Token = the RecordGod store token already minted.

2. New-stock approval landing (staff confirm release_id before commit).

• A /admin "New stock" review view over staged rows that came from distro ingest: shows the scraped title + barcode + auto-resolved release_id (with cover) + the cost; staff confirm or re-pick the release, set retail price, then publish (staged→live). Resolve heuristics John named: almost always reissues → newest year; assume black vinyl unless the slug/title says coloured; barcode pins it when present.

3. New/used in the inventory UI. Add condition_type (new/used toggle) + cost_price to the inventory edit form + show on the list; surface margin (price − cost) where both exist.

4. Wishlist scarcity (phase 2, speculative). Map wishlist rows → release_id, then count how many stores / Discogs sellers stock each (DealGod has the cross-store + Discogs-seller data; reach via its API). Shows relative scarcity to prioritise buys. Needs the Klevu row→product decode first.

5. Other distros (Rocket / Inertia / Bertus). They send invoices (PDF/CSV) not web pages → an invoice ingest (upload → parse line items → same /admin/intake/distro core). Generic source + cost_source already support it. Likely an LLM/parse step per invoice format.

Open questions for John

• Order URL pattern: is it rarewaves.com/account/orders/<id> (new accounts) — confirm so the content-script match is right.
• Retail price on distro stock: auto-suggest from DealGod median, or staff set per item at approval?
• Wishlist scarcity: worth the Klevu-decode effort now, or park until orders+cost are proven in daily use?