MRPGI/docs/HANDOVER.md

# MRPGI — Handover: making the engine fully programmatically controllable

> **Status:** ✅ **BUILT** (2026-07-05) — P0–P6 all landed: `mrpgi-core` workspace carve,
> `GameState` + Command/Event bus, headless JSONL CLI + `--script` replay +
> `--render-room`, game folders (`game.json`), data-driven verbs, flags &
> dialogue gating, HTTP + MCP surfaces, audio events with per-game file
> overrides, headless editor commands. See [CONTROL.md](CONTROL.md) for the
> control-surface guide. This document is kept as the design rationale.
> **Date:** 2026-07-05
> **Purpose:** capture the vision, the current-state analysis, the decisions locked in, and the phased plan so any future session (human or AI) can pick up exactly where we left off.

---

## 1. The dream (in the owner's words)

> "It's an engine for a text adventure but it's got its own paint app inside it and can import art or images and you can assign functions to bands or objects, so you can build a Sierra-style game with it. I want it **100% fully programmatically controllable via CLI / terminal / endpoints / hooks / websockets / MCP / Python**, so an AI can create a whole game from prompts and I can just drop JSON text lore in for expansions. As many features as the OG engine had, plus newer tech. Use **Strudel** to make sounds, which can also be controlled. Always want **options**."

The engine is **MRPGI — Monster Robot Party Game Interpreter**: a new-school, Apple-Silicon-native reimagining of Sierra's 1980s AGI engine (King's Quest / Space Quest / Police Quest), written in Rust + [macroquad](https://macroquad.rs).

---

## 2. Decisions locked this session

| Question | Decision |
|---|---|
| Primary control surfaces | **MCP server + CLI/stdio (JSONL) + HTTP/WebSocket** (all three). Python rides on HTTP. |
| Runtime model | **Headless-first, GUI optional.** The engine core runs with no window; macroquad becomes one optional viewer. |
| Audio | **Offer all backends, toggle any combination on/off** — native chiptune synth, audio files (mp3/ogg/wav), and Strudel. |
| Guiding principle | **"Always want options."** Every subsystem is a swappable, runtime-toggleable backend; nothing is locked in. |

---

## 3. Architecture at a glance

![MRPGI target architecture](architecture.svg)

```mermaid
flowchart TD
    subgraph FE["Front-ends — how you drive it (any/all at once)"]
        GUI["macroquad GUI<br/>play + paint · optional"]
        CLI["CLI · stdio<br/>JSONL in, events out"]
        HTTP["HTTP + WebSocket<br/>REST · SSE · Python"]
        MCP["MCP server<br/>Claude authors + plays"]
    end

    BUS(["Command / event bus<br/>commands down · events up"])

    subgraph CORE["mrpgi-core — headless engine (zero macroquad)"]
        GS["GameState<br/>apply(cmd) → events · tick(dt) → events"]
        PARSE["Parser + AI lane<br/>verb·noun → authoritative logic"]
        FB["FrameBuffer<br/>visual + priority buffers"]
        WORLD["World<br/>rooms · exits · flags/vars"]
        RENDER["render_visual()<br/>→ PNG, no window"]
    end

    DATA["A game = a folder<br/>game.json · rooms/*.json · sprites/*.png · audio"]
    AUDIO["Audio out — toggle both/either/off<br/>SynthSink · FileSink (mp3/ogg/wav) · StrudelSink"]

    GUI --> BUS
    CLI --> BUS
    HTTP --> BUS
    MCP --> BUS
    BUS --> CORE
    DATA -->|load_game · lore upsert| CORE
    CORE -->|AudioSink events| AUDIO
```

**The one insight:** "CLI or terminal or endpoints or hooks or websockets or MCP or Python" is **one feature, not seven** — a headless core behind a typed command/event bus. Every surface is a thin adapter that pushes `Command`s in and reads `Event`s out.

---

## 4. Current-state assessment (the honest read)

MRPGI is a solid, coherent ~3,500-line AGI recreation whose foundations are **much closer to headless-ready than they look.**

**What's genuinely good and reusable as-is:**
- The real AGI priority/depth trick: dual `visual` + `priority` buffers, a `band(y)` depth table, and per-pixel `pri >= room.priority[i]` compositing (`compose()` at `src/main.rs:387`). This is the actual "walk behind the tree" mechanism, not a fake z-sort.
- A deterministic verb-noun parser (`run_command()` at `src/main.rs:586`) with an **AI fallback that cannot cheat** — the LLM only translates free text into an *already-allowed* command, which is then re-run through the real logic (`src/ai.rs`). This is the correct architecture; most homages let the LLM invent state.
- A 1,362-line in-engine paint editor with dual LOOK + MEANING layers, so art and walkability/depth are painted together (`src/editor.rs`).
- **The whole data model already derives `serde`** with disciplined `#[serde(default)]`: `RoomDoc`, `ObjDef`, `DlgNode`, `Stroke`, `Meaning` (`src/editor.rs:104-146`). Rooms already round-trip to `rooms/roomN.json`; there is already a `kit/` of themed objects and locations.

**The single weakness — it is structural, not rot:**
`src/main.rs` fuses three things into one macroquad loop:
1. **game state** — locals at `src/main.rs:68-76` (`ego`, `inventory`, `taken`, `won`, `dlg`, `command`, `transcript`);
2. **update logic** — `ego.update()` (283), `run_command()` (188), the exit-transition scan (289–324), the AI poll (234);
3. **presentation** — `get_frame_time()`, `get_char_pressed()`, `mouse_position()`, `compose()`, `draw_texture_ex()`, `next_frame().await`.

There is currently **no way to advance the game without rendering a frame.** Everything the owner wants is downstream of fixing that one fusion.

**Why it's ~80% ready and doesn't know it:** `ScreenObj::update(&FrameBuffer)`, the entire `picture::*` / `FrameBuffer` / `Cel` / `PicOp` pipeline, and `run_command()` are **already pure computation with no macroquad dependency.** They already take `&mut FrameBuffer` and return values. This is a *give-it-a-spine* job, not a rewrite.

---

## 5. The core architectural move

Split the crate into a Cargo workspace:
- **`mrpgi-core`** — a library with **zero** macroquad dependency (a compile error is the guardrail).
- **`mrpgi`** — the macroquad GUI, now just *one consumer* of the core.

Hoist the scattered `main()` locals into one owned struct with two pure methods and a typed bus:

```rust
struct GameState {
    ego: ScreenObj,
    world: World,          // rooms + current + FrameBuffer (today: the Editor struct)
    inventory: Vec<String>,
    taken: Vec<(u32, String)>,
    won: bool,
    dlg: Option<Dlg>,
    transcript: Vec<String>,
    flags: HashMap<String, bool>,   // P4
    vars:  HashMap<String, i32>,    // P4
}

impl GameState {
    fn apply(&mut self, cmd: Command) -> Vec<Event>;  // wraps run_command, goto_room, move_target, paint
    fn tick(&mut self, dt: f32)     -> Vec<Event>;     // wraps ego.update + exit-transition scan
}
```

- `compose()` splits into a pure `render_visual(&self) -> Vec<u8>` (lives in core) + a 3-line macroquad texture blit (lives in the GUI).
- The clock becomes a `dt: f32` **parameter** instead of `get_frame_time()`, so scripted replays are bit-identical.
- Input becomes `Command`s pushed onto the bus instead of direct `is_key_down` reads.
- **The AI lane needs no change** — it already runs on a worker thread and reports over an `mpsc` channel (`src/ai.rs:81-88`), which is the event-bus pattern in miniature. It just moves from a `main.rs` local into the engine.

### Command / Event sketch (starting point)

```rust
enum Command {
    Walk { dir: u8 }, MoveTo { x: i32, y: i32 },
    Parse { text: String }, Verb { verb: String, noun: String },
    GotoRoom { n: u32 },
    LoadRoom { n: u32, doc: RoomDoc }, UpsertObject { room: u32, obj: ObjDef },
    PaintStroke { stroke: Stroke },
    SetMusic { mood: u8 }, PlaySfx { cue: String },
    NewGame, Query,
}

enum Event {
    Transcript(String), RoomChanged(u32), InventoryChanged,
    DialoguePrompt { node: usize }, Won,
    Audio(AudioEvent), FrameReady, Error(String),
}
```

---

## 6. Control surfaces (all thin adapters over the bus)

| Surface | How it works | Effort |
|---|---|---|
| **In-process Rust API** | `Engine::apply(Command) -> Vec<Event>` — the substrate everything wraps. | M |
| **CLI (stdio, JSONL)** | `mrpgi headless` reads newline JSON commands on stdin, writes newline JSON events on stdout. `--script foo.jsonl` replays a command log for deterministic tests. | S |
| **HTTP + WebSocket** | axum listener on a worker thread (exactly like `ai.rs` spawns its translate thread). `POST /command`, `GET /state`, `GET /events` (SSE), `GET /frame.png` (headless render), `POST /rooms/{n}`, `POST /games`. | M |
| **MCP server** | Wrap the bus as MCP tools: `mrpgi_apply_command`, `mrpgi_get_state`, `mrpgi_load_room`, `mrpgi_paint_stroke`, `mrpgi_render_frame` (returns the image so the model can *see* its own moves), `mrpgi_new_game`. | M |
| **Python** | Thin client over the HTTP server (or a pyo3 binding). Falls out for free. | S |

**Concurrency model:** prefer a single owner thread with an `mpsc` command channel (mirroring `ai.rs`) over a `Mutex<Engine>` — keeps tick ordering deterministic, avoids lock contention.

---

## 7. Audio — "always want options"

Model audio as a `trait AudioSink` behind the bus. Sound calls (currently imperative: `sfx.pickup()` at `src/main.rs:162,199,217-224`) become `Event::Audio(..)` on the same event bus every control surface already consumes. Then pick sink(s) at runtime and/or declare them per-game in `game.json`:

- **`SynthSink`** — the existing `src/sound.rs` runtime chiptune synth. Offline default, procedural, tiny, LLM-authorable as text.
- **`FileSink`** — plays `.mp3` / `.ogg` / `.wav` from the game folder. macroquad's `load_sound` auto-detects all three (`load_sound_from_bytes` is already what `sound.rs` calls, so this is a data-source swap, not a new system). **Prefer OGG for looping music** — MP3 encoders pad a few ms of silence that clicks at loop seams; fine for one-shot SFX. Likely the **first** audio backend to build (easier than the Strudel bridge).
- **`StrudelSink`** — Strudel is browser/Web-Audio JS; the engine is native Rust, so they meet at a **message boundary, not shared memory.** The engine emits audio *descriptors* (`{sfx:"door"}`, `{pattern:"note(\"c e g\").s(\"sawtooth\")"}`) over the WebSocket to a tiny browser page running `@strudel/core` + `@strudel/webaudio`, which is the actual synth. Rooms gain an optional `music_pattern: Option<String>` so an LLM-authored room can carry a Strudel one-liner in its JSON.

Any combination runs at once, or none (silent). This mirrors the existing `ai.rs` provider switch (`ollama / openrouter / off`) — the template for every option in the engine.

**The rule that keeps options from becoming chaos:** options live at the *edges*, never in the core. `GameState` neither knows nor cares which sink or control surface is attached — it only emits `Event`s and applies `Command`s. That's what makes "add another option later" a 30-line trait impl instead of a rewrite.

---

## 8. Games as data (the "drop JSON lore in" milestone)

The bones already exist (`RoomDoc`, `ObjDef`, `DlgNode`, `Stroke` all serde; rooms round-trip; `kit/` holds JSON archetypes). The gap: a "game" is not yet a first-class loadable unit — the engine hardcodes `rooms/roomN.json` paths, room 0 as boot, and the seven verbs in code.

1. **Game manifest** — a `GameManifest` (`game.json`): `{ name, start_room, start_spawn, rooms_dir, sprites_dir, verbs, intro_text, win_text, flags, music_patterns }`. A game becomes a folder: `games/<name>/{game.json, rooms/*.json, sprites/*.png, sfx/*, music/*}`. `World::load_game(dir)` replaces the hardcoded paths.
2. **Data-driven verbs** — lift the verb table (today split across `canon_verb` `src/main.rs:551`, `build_prompt` `src/ai.rs:73`, and the `run_command` match) into `Vec<VerbDef>` in the manifest. `build_prompt` then generates its whitelist *from* this table, so the AI and the parser can never drift.
3. **Flags & vars** — add `flags`/`vars` to `GameState`; give `ObjDef`/`DlgChoice` optional `requires`/`sets` (all `#[serde(default)]` so existing rooms keep loading). Turns one-shot `wins` and tree dialogue into real gated puzzles.
4. **Lore ingestion** — because the types are serde, an LLM expansion is just JSON that deserializes. Add `UpsertRoom` / `UpsertObject` / `PatchDialogue` commands so a running game extends live over HTTP/MCP. Ship a schemars-derived **JSON Schema** as the authoring contract. Validate ingested rooms with the existing `find_spawn` (`src/editor.rs`) + exit-target resolution before committing.

---

## 9. AGI-soul gaps surfaced (real, but downstream of the core move)

Not on the critical path to programmability — sequence *after* the bus/server land, or the refactor never ships:
- Only one active room at a time (`editor.current: u32`) — no cross-room NPCs/state.
- Click-to-walk uses naive greedy steering (`sprite.rs`), no pathfinding — gets stuck on diagonal obstacles.
- Dialogue is tree-only, no conditions on choices.
- Victory is a single `won: bool` (no flags/vars) — addressed in P4.
- No item combining; take/drop are simple vec push/remove.
- Room dimensions hardcoded 160×168 (`framebuffer.rs`).

---

## 10. Phased roadmap (each phase ships something usable)

| Phase | Goal | Unlocks |
|---|---|---|
| **P0 · Carve the core** | Workspace: `mrpgi-core` (lib, no macroquad) + `mrpgi` (GUI). Move the pure pieces over. No behaviour change. | A compilable proof the sim has no rendering dependency — precondition for everything. |
| **P1 · GameState + bus** | `GameState` with `apply(Command)`/`tick(dt)`; GUI rewritten to drive them; `compose()` split; AI worker delivers a `Command::Parse` result as an `Event`. | Engine fully drivable & observable in-process without rendering. Golden-transcript tests possible. |
| **P2 · Headless CLI + game folders** | `mrpgi headless` (JSONL, `--script` replay); `GameManifest` + `World::load_game`; data-driven verb table. | LLM/pipe control today; CI golden tests; authoring a game without touching Rust. **("Drop JSON lore in.")** |
| **P3 · HTTP/WS + MCP** | axum server (incl. headless `/frame.png`); MCP tools incl. `render_frame`; Python client; JSON Schema authoring contract. | Claude/Python/curl drive **and** author games; the model can *see* frames it produces. **The headline scenario.** |
| **P4 · AGI soul: flags/vars/conditions** | `flags`/`vars` in `GameState`; `requires`/`sets` on objects & dialogue; upsert commands with validation. | Real puzzles, gated dialogue, multi-step quests — all authorable as JSON. |
| **P5 · Audio sinks** | `trait AudioSink`; `sound.rs` → `SynthSink`; `FileSink` (mp3/ogg/wav) + `music/`+`sfx/` folder convention; `StrudelSink` browser sidecar over WS; `music_pattern` on rooms. | Programmatic, LLM-authored music/sfx; every backend toggleable. |
| **P6 · Headless editor** | `PaintStroke`/`UpsertObject` commands so the editor's stroke pipeline runs without `frame()`; GUI editor becomes a thin front-end over the same commands. | An LLM paints rooms via the same bus a human uses. |

---

## 11. Quick wins (afternoon-sized, near-zero risk — the recommended first move)

1. **`mrpgi --render-room rooms/room0.json out.png`** — bake a `RoomDoc` to PNG headless via the existing compose path. Proves headless rendering with almost no refactor; gives the LLM a way to preview authored rooms *today*.
2. **Lift `run_command()` into `mrpgi-core` and unit-test it** with golden `(input → lines, understood)` cases. Zero risk; instant safety net for the P1 refactor.
3. **Generate the AI verb list in `build_prompt` (`src/ai.rs:68`) *from* `canon_verb`'s table** instead of a hardcoded string, so the two can never drift. ~15 min; also preps data-driven verbs.
4. **Wrap sound calls behind `trait AudioSink`** with the existing `Sfx`/`Music` as the impl (no behaviour change). Mechanical; unlocks the whole audio-options path later.

---

## 12. Risks & guardrails

- **macroquad owns the window/audio/GL context** via `#[macroquad::main]`; the core must never call into it. Enforce with the workspace boundary — a compile error is the guardrail. Watch `assets.rs`: the `quantize()`/`Cel` path is pure (`image` crate) but any macroquad `Image`/`Texture` use stays in the front-end.
- **Determinism:** the headless core must take `dt` explicitly and fix the cycle rate (`CYCLE_DT = 1/20`) so replays are bit-identical. Keep the (non-deterministic) AI lane *out* of the deterministic core path — it produces a `Command` that then runs through deterministic logic (the current design already respects this).
- **Backward-compat:** every new `RoomDoc`/`ObjDef` field MUST be `#[serde(default)]` (the codebase already does this) so the existing `rooms/*.json` and `kit/` packs keep loading.
- **Strudel is a genuine native/web split** — accept a browser sidecar as the real path; don't over-invest in a native mini-notation interpreter until a native-only user needs it.
- **Editor/Play share state** — the core split is the chance to make `World` the single source of truth. Keep the GUI editor driving the *same* commands the server uses, rather than a parallel path, to avoid regressing the working editor.

---

## 13. Open questions / next step

**Recommended opening play:** P0 (carve `mrpgi-core`) + the four quick wins, on a branch — gets a headless engine rendering a room and passing its first tests without touching the fun stuff.

**Or keep designing first:** nail down the exact `Command`/`Event` enums and the `game.json` manifest schema before any code moves.

Still to decide when building starts:
- HTTP framework: axum vs. a minimal hyper/tiny_http (weigh against build times).
- MCP transport: stdio (simplest) to start.
- Whether `FileSink` lands in P2 (alongside game folders) or waits for P5.

---

## 14. Source map (current engine)

| File | Lines | Role |
|---|---|---|
| `src/main.rs` | 728 | Window, game loop, mode switch, parser (`run_command`), compositing, HUD. **The fusion to break up.** |
| `src/editor.rs` | 1362 | In-engine paint tool; also the de-facto world database (`RoomDoc`, `ObjDef`, `DlgNode`, sprite catalog). |
| `src/assets.rs` | 440 | PNG → EGA quantize + dither; tile / pour-into-shape / object insertion; hot-reload. |
| `src/picture.rs` | 283 | Vector room painting (`PicOp` rects / lines / pixels), `stamp_cel_fit`. |
| `src/sprite.rs` | 166 | The ego: movement, click-to-walk, collision, walk animation; `Prop`. |
| `src/ai.rs` | 134 | AI parser lane (Ollama / OpenRouter / off) — **the template for the whole option/adapter pattern.** |
| `src/sound.rs` | 151 | Runtime square-wave chiptune synth + 5 music moods → WAV bytes. |
| `src/framebuffer.rs` | 87 | The `visual` + `priority` buffers and the `band(y)` depth table. |
| `src/room.rs` | 72 | Hand-authored demo `Scene`. |
| `src/view.rs` | 51 | Sprites as loops + cels, authored from ASCII grids. |
| `src/palette.rs` | 40 | The 16-color EGA palette (indices → RGBA). |

Resources: `rooms/*.json` (room docs), `kit/` (themed objects + locations — proto game-folders), `sprites/*.png`, `assets/*.png`, `palettes/`. Docs: `README.md`, `MANUAL.md`, `EDITOR_GUIDE.md`.

---

## 15. Provenance

This handover was produced from a full read of the engine source (`src/*.rs`) plus a parallel architecture-mapping pass over every subsystem (game loop, editor, render pipeline, actors, AI + sound, resources) and a synthesis design pass. It reflects the state of `main @ d0caf98` (Initial commit) and the design conversation of 2026-07-05. Diagram source: [`architecture.svg`](architecture.svg).