diff --git a/RAREWAVES_INGEST_PLAN.md b/RAREWAVES_INGEST_PLAN.md new file mode 100644 index 0000000..590c5cd --- /dev/null +++ b/RAREWAVES_INGEST_PLAN.md @@ -0,0 +1,61 @@ +# Distro ingest (RareWaves first) + cost tracking + new/used — plan (2026-06-27) + +Buy stock from distros (RareWaves discount sales, ~$8–15 capped shipping → margin in volume), scrape the +**order** + **wishlist** account pages via the PRICEGOD extension, land **cost price per copy** in inventory, +and use the **wishlist** to gauge scarcity (how many stores / Discogs sellers stock it). Prep Rocket / Inertia / +Bertus next (they send invoices to ingest). + +## ✅ Done (foundations, live on recordgod) +- `inventory.cost_price numeric(10,2)` — per-COPY buy price (costs vary per copy, so inventory-level not release-level). +- `inventory.cost_source text` — provenance, e.g. `rarewaves #592619`. +- `inventory.condition_type text NOT NULL DEFAULT 'used'` — the **new/used binary split**. Backfilled: 451 sealed + items (`notes ILIKE '%seal%'`) → `new`; everything else `used`. (Distro-bought stock defaults `new`.) + +## What the RareWaves dumps revealed (analysis) +RareWaves = **Shopify**. Source files on ultra `~/Documents/recordgod/rarw*.txt`. + +**Orders (rarworder*.txt)** — new Shopify customer-account React pages. Each line item: +``` + + Rumours … quantity 4 … $42.99/ea +``` +→ **barcode (EAN-13 = leading digits of the handle)** + **title (img alt)** + **year (in slug)** + **qty** + +**unit cost (`$X.XX/ea`)** + variant id + order # (`Order #592619`). CSS classes are obfuscated hashes — key off +the **`/products/-` URL**, the **`/ea` price text**, ARIA `role="row"`, and `quantity N`. **High confidence.** + +**Wishlist (rarwwish*.txt)** — rendered by the **Klevu** app into collapsible `data-wishlist-row-id` rows. No clean +barcode in the dump (the 13-digit hits were CDN cache-bust hashes `?v=…`, false positives). **Harder** — needs +row→product mapping, likely via Klevu's data or by following each row's product link. Lower confidence; phase 2. + +## Build plan (sequenced) + +**1. Order scraper → cost into inventory (HIGH value, HIGH confidence — do first).** +- PRICEGOD content script on `rarewaves.com/account/orders/*` (+ legacy `/orders/*`): walk each `role="row"`, + pull `{barcode, title, year(slug), qty, unit_cost, variant_id}` + the order number. +- New RecordGod endpoint `POST /admin/intake/distro` `{source:'rarewaves', order_ref, items:[…]}` → per item: + resolve **barcode → release_id** (local `disc_release_identifier` `type='Barcode'`, **normalised**: strip + non-digits, try with/without leading 0; Discogs `/database/search?barcode=` fallback), enrich, **stage** with + `cost_price=unit_cost`, `cost_source='rarewaves #'`, `condition_type='new'`, one SKU per copy (qty → N rows). +- Reuses the existing `_stage`/`_enrich` + the staged review queue. Token = the RecordGod store token already minted. + +**2. New-stock approval landing (staff confirm release_id before commit).** +- A `/admin` "New stock" review view over `staged` rows that came from distro ingest: shows the scraped title + + barcode + auto-resolved release_id (with cover) + the cost; staff **confirm or re-pick** the release, set + retail price, then **publish** (staged→live). Resolve heuristics John named: almost always **reissues → newest + year**; assume **black vinyl** unless the slug/title says coloured; barcode pins it when present. + +**3. New/used in the inventory UI.** Add `condition_type` (new/used toggle) + `cost_price` to the inventory +edit form + show on the list; surface **margin** (price − cost) where both exist. + +**4. Wishlist scarcity (phase 2, speculative).** Map wishlist rows → release_id, then count **how many stores / +Discogs sellers** stock each (DealGod has the cross-store + Discogs-seller data; reach via its API). Shows relative +scarcity to prioritise buys. Needs the Klevu row→product decode first. + +**5. Other distros (Rocket / Inertia / Bertus).** They send **invoices** (PDF/CSV) not web pages → an invoice +ingest (upload → parse line items → same `/admin/intake/distro` core). Generic `source` + `cost_source` already +support it. Likely an LLM/parse step per invoice format. + +## Open questions for John +- Order URL pattern: is it `rarewaves.com/account/orders/` (new accounts) — confirm so the content-script match is right. +- Retail price on distro stock: auto-suggest from DealGod median, or staff set per item at approval? +- Wishlist scarcity: worth the Klevu-decode effort now, or park until orders+cost are proven in daily use? diff --git a/app/main.py b/app/main.py index 2f92912..c754ad2 100644 --- a/app/main.py +++ b/app/main.py @@ -106,6 +106,10 @@ _STARTUP_DDL = [ "ALTER TABLE staff ADD COLUMN IF NOT EXISTS pay_rate numeric(10,2)", "ALTER TABLE staff ADD COLUMN IF NOT EXISTS password_hash text", "CREATE UNIQUE INDEX IF NOT EXISTS staff_email_uniq ON staff (lower(email)) WHERE email IS NOT NULL", + # cost tracking (per-copy buy price from distro orders) + new/used split + "ALTER TABLE inventory ADD COLUMN IF NOT EXISTS cost_price numeric(10,2)", + "ALTER TABLE inventory ADD COLUMN IF NOT EXISTS cost_source text", # e.g. 'rarewaves #592619' + "ALTER TABLE inventory ADD COLUMN IF NOT EXISTS condition_type text NOT NULL DEFAULT 'used'", # 'new' | 'used' ]