07a0db6b61
Distill pattern: embedded forum corpus (style reference only, never shipped verbatim) → persona line banks with human review gate → SFT export for LoRA. First deployment: PROCITY shopkeepers from Soulstrut/VGplus on ultra. Includes the abliterated writer toggle and the earned-gotchas writeup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
138 lines
8.0 KiB
Markdown
138 lines
8.0 KiB
Markdown
# NPCvector
|
||
|
||
**Turn any scraped text corpus + pgvector into a cast of game NPCs with authentic
|
||
voices — without ever shipping a word of the original corpus.**
|
||
|
||
First deployment: **GABGOD** on ultra — Soulstrut (818k embedded posts) + VeryGoodPlus
|
||
(59k) forum corpus → sarcastic shopkeepers for PROCITY's 90s Australian shopping town.
|
||
But the pattern is generic: any corpus you can embed, any cast of characters, any game.
|
||
|
||
```
|
||
THE PATTERN
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ corpus (pgvector) personas.json │
|
||
│ forum posts, reviews, who talks, how they talk, │
|
||
│ transcripts, chat logs what situations they're in │
|
||
│ │ │ │
|
||
│ └───────► distill.py ◄─────┘ │
|
||
│ 1. embed the situation (nomic-embed-text, 768d) │
|
||
│ 2. pgvector top-k similar corpus posts = STYLE REFERENCE │
|
||
│ 3. writer LLM (toggleable, see below) writes ORIGINAL lines │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ line_bank table (approved = NULL until a human reviews) │
|
||
│ │ │
|
||
│ ├──► in-game retrieval dialogue (line bank + small │
|
||
│ │ embedding model in the browser via LiteRT.js — │
|
||
│ │ player types anything, semantic match picks the │
|
||
│ │ NPC's comeback; tiny download; fully offline) │
|
||
│ │ │
|
||
│ └──► export_sft.py → SFT JSONL → LoRA a local model │
|
||
│ (bake the VOICE into a model; facts stay in RAG) │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Why distill instead of RAG-ing the corpus straight into the game?
|
||
|
||
1. **Real people wrote the corpus.** Verbatim forum posts coming out of an NPC's
|
||
mouth in a public game is a legal grey zone and recognizably *someone*. The
|
||
corpus is used as **voice and attitude reference only**; the prompt forbids
|
||
copying, and provenance (`source_post_ids`) is kept so any line can be audited
|
||
against its references.
|
||
2. **Review gate.** Every generated line lands with `approved = NULL`. A human
|
||
eyeballs before anything ships (house rule — same as mocapgod QC clips).
|
||
Essential when the writer model is uncensored (below).
|
||
3. **Tiny runtime.** A reviewed line bank + a small embedding model runs in the
|
||
browser (LiteRT.js / WASM / WebGPU) — no server, no 2GB model download for
|
||
players, works offline on monsterrobot.games.
|
||
|
||
## The abliterated toggle
|
||
|
||
```bash
|
||
./gabgod abliterated status # which writer distill.py will use
|
||
./gabgod abliterated on # uncensored writer (local, private generation)
|
||
./gabgod abliterated off # stock writer (safe default)
|
||
```
|
||
|
||
`config.json → writer.abliterated` flips between two OpenAI-compatible endpoints:
|
||
|
||
| | model | where | when |
|
||
|---|---|---|---|
|
||
| **off** | qwen2.5:7b | m4pro Ollama | safe default, boring but obedient |
|
||
| **on** | supergemma4-26b-uncensored (16GB GGUF) | ultra's own Ollama | commits to the bit, no refusals |
|
||
|
||
(First choice was gemma-4-E4B-it-OBLITERATED — abandoned after it drifted into
|
||
Chinese mid-run and failed 25/40 topics. Abliteration damages small models;
|
||
the 26B holds together. See "Earned gotchas" below.)
|
||
|
||
**House rule: abliterated output never ships publicly unreviewed.** The toggle
|
||
gates *generation*; the `approved` column gates *shipping*. Both must be green
|
||
for a line to reach a public game. For private/LAN free-chat NPCs, point the
|
||
game at the abliterated endpoint directly and skip the bank — your house, your
|
||
rules.
|
||
|
||
## Using it on something else later (the reusable bit)
|
||
|
||
1. **Get a corpus into pgvector.** Any table with `(id, body_text, embedding vector(768))`
|
||
embedded with `nomic-embed-text` works. Add the table name to
|
||
`config.json → retrieval.sources`. (In discogs_full you already have embedded:
|
||
soulstrut_post, vgplus_post, ishkur_style, discogs_reviews 414k, yt_transcript,
|
||
release_wikipedia 1.3M, artist_wikipedia, semantic_gold…)
|
||
2. **Write personas.json.** Per character: `voice` (who they are, how they talk)
|
||
and `topics` (situations they'll need lines for). The topics double as the
|
||
retrieval queries — write them in the corpus's vocabulary for better matches.
|
||
3. **Run `./gabgod distill`.** Heartbeat in `~/.jobs/gabgod-distill.status`,
|
||
resume-safe (persona-topics with enough lines are skipped on rerun).
|
||
4. **Review** (`./gabgod sample <persona>`, then set `approved` in psql/pgweb),
|
||
**export** (`./gabgod export` → `sft/train.jsonl` + `valid.jsonl`), and either
|
||
ship the bank or LoRA the voice (`mlx_lm.lora --data sft/ ...` on m3ultra).
|
||
|
||
Ideas already on the list: RECORDGOD store-assistant voice from discogs_reviews;
|
||
MRPGI narrator personas; toastsim's judge from the same forums; Ishkur-flavoured
|
||
genre-pedant NPC from ishkur_style (153 embedded style blurbs).
|
||
|
||
## Files
|
||
|
||
| file | what |
|
||
|---|---|
|
||
| `distill.py` | the forge: retrieve → write → insert. `--persona X --topics N --dry-run` to smoke-test |
|
||
| `personas.json` | the cast (currently: PROCITY's six shopkeepers) |
|
||
| `config.json` | endpoints + toggle + retrieval knobs (gitignored; start from `config.example.json`) |
|
||
| `schema.sql` | `gabgod.line_bank` (+ schema) — run once per database |
|
||
| `export_sft.py` | approved lines → chat-format JSONL for LoRA |
|
||
| `gabgod` | CLI: `abliterated on/off/status · distill · status · sample · counts · export` |
|
||
|
||
## Ops notes
|
||
|
||
- Deployment lives on **ultra** `~/Documents/GABGOD` (the data's there; scripts use
|
||
`/opt/homebrew/bin/python3` per TCC rule). This repo is the canonical source.
|
||
- Long runs follow the heartbeat convention (`~/.jobs/`, overwrite-not-append,
|
||
die loudly, checkpoint+resume).
|
||
- Embeddings: nomic-embed-text (768d) — **must match whatever embedded the corpus**,
|
||
or retrieval silently degrades. Check dims with `atttypmod` on the vector column.
|
||
- Postgres gotcha that bit here: `n_live_tup` stats on the corpus tables read 0
|
||
(stale autovacuum); trust `count(*)`, not `pg_stat_user_tables`.
|
||
- Ollama structured-output note: small models ignore "exactly N lines" — the
|
||
parser takes whatever valid JSON array comes back (raw_decode from first `[`).
|
||
|
||
## Earned gotchas (one bad evening's tuition)
|
||
|
||
1. **Abliterated small models are brain-damaged.** The E4B weight-edit produced
|
||
Chinese, word salad, and malformed JSON under a real workload despite passing
|
||
a simple smoke test. Test with the PRODUCTION prompt, not "say 3 phrases".
|
||
2. **Reasoning models + long prompts = empty answers.** supergemma via the
|
||
OpenAI-compat `/v1` endpoint burned its entire 4096-token budget on internal
|
||
drafts (`finish_reason: length`, 12k chars of reasoning, zero content). Fix:
|
||
use ollama's native `/api/chat` with `"think": false` — 12/12 lines in 9s vs
|
||
nothing in 5 minutes. distill.py falls back automatically for models with no
|
||
think switch.
|
||
3. **Keep the reference block small.** >6 refs × 400 chars measurably worsens
|
||
deliberation spirals; retrieval `top_k` can stay high, but cap what you show
|
||
the writer.
|
||
4. **`sane_line()` is load-bearing.** Language-drift and fragment junk WILL
|
||
reach the DB without the ascii-ratio/length gate; purging after the fact is
|
||
`delete ... where length(regexp_replace(line,'[[:ascii:]]','','g')) > 3`.
|
||
5. **Resume counts must be per-writer-model** (`writer_model=%s` in the skip
|
||
query), or a failed model's junk "completes" topics and blocks the good
|
||
model from refilling them.
|