MRPGI/docs/HANDOVER.md

MRPGI — Handover: making the engine fully programmatically controllable

Status: design / discussion complete. No engine code has been changed yet. Date: 2026-07-05 Purpose: capture the vision, the current-state analysis, the decisions locked in, and the phased plan so any future session (human or AI) can pick up exactly where we left off.

---

1. The dream (in the owner's words)

"It's an engine for a text adventure but it's got its own paint app inside it and can import art or images and you can assign functions to bands or objects, so you can build a Sierra-style game with it. I want it 100% fully programmatically controllable via CLI / terminal / endpoints / hooks / websockets / MCP / Python, so an AI can create a whole game from prompts and I can just drop JSON text lore in for expansions. As many features as the OG engine had, plus newer tech. Use Strudel to make sounds, which can also be controlled. Always want options."

The engine is MRPGI — Monster Robot Party Game Interpreter: a new-school, Apple-Silicon-native reimagining of Sierra's 1980s AGI engine (King's Quest / Space Quest / Police Quest), written in Rust + macroquad.

---

2. Decisions locked this session

Question	Decision
Primary control surfaces	MCP server + CLI/stdio (JSONL) + HTTP/WebSocket (all three). Python rides on HTTP.
Runtime model	Headless-first, GUI optional. The engine core runs with no window; macroquad becomes one optional viewer.
Audio	Offer all backends, toggle any combination on/off — native chiptune synth, audio files (mp3/ogg/wav), and Strudel.
Guiding principle	"Always want options." Every subsystem is a swappable, runtime-toggleable backend; nothing is locked in.

---

3. Architecture at a glance

flowchart TD
    subgraph FE["Front-ends — how you drive it (any/all at once)"]
        GUI["macroquad GUI<br/>play + paint · optional"]
        CLI["CLI · stdio<br/>JSONL in, events out"]
        HTTP["HTTP + WebSocket<br/>REST · SSE · Python"]
        MCP["MCP server<br/>Claude authors + plays"]
    end

    BUS(["Command / event bus<br/>commands down · events up"])

    subgraph CORE["mrpgi-core — headless engine (zero macroquad)"]
        GS["GameState<br/>apply(cmd) → events · tick(dt) → events"]
        PARSE["Parser + AI lane<br/>verb·noun → authoritative logic"]
        FB["FrameBuffer<br/>visual + priority buffers"]
        WORLD["World<br/>rooms · exits · flags/vars"]
        RENDER["render_visual()<br/>→ PNG, no window"]
    end

    DATA["A game = a folder<br/>game.json · rooms/*.json · sprites/*.png · audio"]
    AUDIO["Audio out — toggle both/either/off<br/>SynthSink · FileSink (mp3/ogg/wav) · StrudelSink"]

    GUI --> BUS
    CLI --> BUS
    HTTP --> BUS
    MCP --> BUS
    BUS --> CORE
    DATA -->|load_game · lore upsert| CORE
    CORE -->|AudioSink events| AUDIO

The one insight: "CLI or terminal or endpoints or hooks or websockets or MCP or Python" is one feature, not seven — a headless core behind a typed command/event bus. Every surface is a thin adapter that pushes Commands in and reads Events out.

---

4. Current-state assessment (the honest read)

MRPGI is a solid, coherent ~3,500-line AGI recreation whose foundations are much closer to headless-ready than they look.

What's genuinely good and reusable as-is:

• The real AGI priority/depth trick: dual visual + priority buffers, a band(y) depth table, and per-pixel pri >= room.priority[i] compositing (compose() at src/main.rs:387). This is the actual "walk behind the tree" mechanism, not a fake z-sort.
• A deterministic verb-noun parser (run_command() at src/main.rs:586) with an AI fallback that cannot cheat — the LLM only translates free text into an already-allowed command, which is then re-run through the real logic (src/ai.rs). This is the correct architecture; most homages let the LLM invent state.
• A 1,362-line in-engine paint editor with dual LOOK + MEANING layers, so art and walkability/depth are painted together (src/editor.rs).
• The whole data model already derives serde with disciplined #[serde(default)]: RoomDoc, ObjDef, DlgNode, Stroke, Meaning (src/editor.rs:104-146). Rooms already round-trip to rooms/roomN.json; there is already a kit/ of themed objects and locations.

The single weakness — it is structural, not rot: src/main.rs fuses three things into one macroquad loop:

1. game state — locals at src/main.rs:68-76 (ego, inventory, taken, won, dlg, command, transcript);
2. update logic — ego.update() (283), run_command() (188), the exit-transition scan (289–324), the AI poll (234);
3. presentation — get_frame_time(), get_char_pressed(), mouse_position(), compose(), draw_texture_ex(), next_frame().await.

There is currently no way to advance the game without rendering a frame. Everything the owner wants is downstream of fixing that one fusion.

Why it's ~80% ready and doesn't know it: ScreenObj::update(&FrameBuffer), the entire picture::* / FrameBuffer / Cel / PicOp pipeline, and run_command() are already pure computation with no macroquad dependency. They already take &mut FrameBuffer and return values. This is a give-it-a-spine job, not a rewrite.

---

5. The core architectural move

Split the crate into a Cargo workspace:

• mrpgi-core — a library with zero macroquad dependency (a compile error is the guardrail).
• mrpgi — the macroquad GUI, now just one consumer of the core.

Hoist the scattered main() locals into one owned struct with two pure methods and a typed bus:

struct GameState {
    ego: ScreenObj,
    world: World,          // rooms + current + FrameBuffer (today: the Editor struct)
    inventory: Vec<String>,
    taken: Vec<(u32, String)>,
    won: bool,
    dlg: Option<Dlg>,
    transcript: Vec<String>,
    flags: HashMap<String, bool>,   // P4
    vars:  HashMap<String, i32>,    // P4
}

impl GameState {
    fn apply(&mut self, cmd: Command) -> Vec<Event>;  // wraps run_command, goto_room, move_target, paint
    fn tick(&mut self, dt: f32)     -> Vec<Event>;     // wraps ego.update + exit-transition scan
}

• compose() splits into a pure render_visual(&self) -> Vec<u8> (lives in core) + a 3-line macroquad texture blit (lives in the GUI).
• The clock becomes a dt: f32 parameter instead of get_frame_time(), so scripted replays are bit-identical.
• Input becomes Commands pushed onto the bus instead of direct is_key_down reads.
• The AI lane needs no change — it already runs on a worker thread and reports over an mpsc channel (src/ai.rs:81-88), which is the event-bus pattern in miniature. It just moves from a main.rs local into the engine.

Command / Event sketch (starting point)

enum Command {
    Walk { dir: u8 }, MoveTo { x: i32, y: i32 },
    Parse { text: String }, Verb { verb: String, noun: String },
    GotoRoom { n: u32 },
    LoadRoom { n: u32, doc: RoomDoc }, UpsertObject { room: u32, obj: ObjDef },
    PaintStroke { stroke: Stroke },
    SetMusic { mood: u8 }, PlaySfx { cue: String },
    NewGame, Query,
}

enum Event {
    Transcript(String), RoomChanged(u32), InventoryChanged,
    DialoguePrompt { node: usize }, Won,
    Audio(AudioEvent), FrameReady, Error(String),
}

---

6. Control surfaces (all thin adapters over the bus)

Surface	How it works	Effort
In-process Rust API	Engine::apply(Command) -> Vec<Event> — the substrate everything wraps.	M
CLI (stdio, JSONL)	mrpgi headless reads newline JSON commands on stdin, writes newline JSON events on stdout. --script foo.jsonl replays a command log for deterministic tests.	S
HTTP + WebSocket	axum listener on a worker thread (exactly like ai.rs spawns its translate thread). POST /command, GET /state, GET /events (SSE), GET /frame.png (headless render), POST /rooms/{n}, POST /games.	M
MCP server	Wrap the bus as MCP tools: mrpgi_apply_command, mrpgi_get_state, mrpgi_load_room, mrpgi_paint_stroke, mrpgi_render_frame (returns the image so the model can see its own moves), mrpgi_new_game.	M
Python	Thin client over the HTTP server (or a pyo3 binding). Falls out for free.	S

Concurrency model: prefer a single owner thread with an mpsc command channel (mirroring ai.rs) over a Mutex<Engine> — keeps tick ordering deterministic, avoids lock contention.

---

7. Audio — "always want options"

Model audio as a trait AudioSink behind the bus. Sound calls (currently imperative: sfx.pickup() at src/main.rs:162,199,217-224) become Event::Audio(..) on the same event bus every control surface already consumes. Then pick sink(s) at runtime and/or declare them per-game in game.json:

• SynthSink — the existing src/sound.rs runtime chiptune synth. Offline default, procedural, tiny, LLM-authorable as text.
• FileSink — plays .mp3 / .ogg / .wav from the game folder. macroquad's load_sound auto-detects all three (load_sound_from_bytes is already what sound.rs calls, so this is a data-source swap, not a new system). Prefer OGG for looping music — MP3 encoders pad a few ms of silence that clicks at loop seams; fine for one-shot SFX. Likely the first audio backend to build (easier than the Strudel bridge).
• StrudelSink — Strudel is browser/Web-Audio JS; the engine is native Rust, so they meet at a message boundary, not shared memory. The engine emits audio descriptors ({sfx:"door"}, {pattern:"note(\"c e g\").s(\"sawtooth\")"}) over the WebSocket to a tiny browser page running @strudel/core + @strudel/webaudio, which is the actual synth. Rooms gain an optional music_pattern: Option<String> so an LLM-authored room can carry a Strudel one-liner in its JSON.

Any combination runs at once, or none (silent). This mirrors the existing ai.rs provider switch (ollama / openrouter / off) — the template for every option in the engine.

The rule that keeps options from becoming chaos: options live at the edges, never in the core. GameState neither knows nor cares which sink or control surface is attached — it only emits Events and applies Commands. That's what makes "add another option later" a 30-line trait impl instead of a rewrite.

---

8. Games as data (the "drop JSON lore in" milestone)

The bones already exist (RoomDoc, ObjDef, DlgNode, Stroke all serde; rooms round-trip; kit/ holds JSON archetypes). The gap: a "game" is not yet a first-class loadable unit — the engine hardcodes rooms/roomN.json paths, room 0 as boot, and the seven verbs in code.

1. Game manifest — a GameManifest (game.json): { name, start_room, start_spawn, rooms_dir, sprites_dir, verbs, intro_text, win_text, flags, music_patterns }. A game becomes a folder: games/<name>/{game.json, rooms/*.json, sprites/*.png, sfx/*, music/*}. World::load_game(dir) replaces the hardcoded paths.
2. Data-driven verbs — lift the verb table (today split across canon_verb src/main.rs:551, build_prompt src/ai.rs:73, and the run_command match) into Vec<VerbDef> in the manifest. build_prompt then generates its whitelist from this table, so the AI and the parser can never drift.
3. Flags & vars — add flags/vars to GameState; give ObjDef/DlgChoice optional requires/sets (all #[serde(default)] so existing rooms keep loading). Turns one-shot wins and tree dialogue into real gated puzzles.
4. Lore ingestion — because the types are serde, an LLM expansion is just JSON that deserializes. Add UpsertRoom / UpsertObject / PatchDialogue commands so a running game extends live over HTTP/MCP. Ship a schemars-derived JSON Schema as the authoring contract. Validate ingested rooms with the existing find_spawn (src/editor.rs) + exit-target resolution before committing.

---

9. AGI-soul gaps surfaced (real, but downstream of the core move)

Not on the critical path to programmability — sequence after the bus/server land, or the refactor never ships:

• Only one active room at a time (editor.current: u32) — no cross-room NPCs/state.
• Click-to-walk uses naive greedy steering (sprite.rs), no pathfinding — gets stuck on diagonal obstacles.
• Dialogue is tree-only, no conditions on choices.
• Victory is a single won: bool (no flags/vars) — addressed in P4.
• No item combining; take/drop are simple vec push/remove.
• Room dimensions hardcoded 160×168 (framebuffer.rs).

---

10. Phased roadmap (each phase ships something usable)

Phase	Goal	Unlocks
P0 · Carve the core	Workspace: mrpgi-core (lib, no macroquad) + mrpgi (GUI). Move the pure pieces over. No behaviour change.	A compilable proof the sim has no rendering dependency — precondition for everything.
P1 · GameState + bus	GameState with apply(Command)/tick(dt); GUI rewritten to drive them; compose() split; AI worker delivers a Command::Parse result as an Event.	Engine fully drivable & observable in-process without rendering. Golden-transcript tests possible.
P2 · Headless CLI + game folders	mrpgi headless (JSONL, --script replay); GameManifest + World::load_game; data-driven verb table.	LLM/pipe control today; CI golden tests; authoring a game without touching Rust. ("Drop JSON lore in.")
P3 · HTTP/WS + MCP	axum server (incl. headless /frame.png); MCP tools incl. render_frame; Python client; JSON Schema authoring contract.	Claude/Python/curl drive and author games; the model can see frames it produces. The headline scenario.
P4 · AGI soul: flags/vars/conditions	flags/vars in GameState; requires/sets on objects & dialogue; upsert commands with validation.	Real puzzles, gated dialogue, multi-step quests — all authorable as JSON.
P5 · Audio sinks	trait AudioSink; sound.rs → SynthSink; FileSink (mp3/ogg/wav) + music/+sfx/ folder convention; StrudelSink browser sidecar over WS; music_pattern on rooms.	Programmatic, LLM-authored music/sfx; every backend toggleable.
P6 · Headless editor	PaintStroke/UpsertObject commands so the editor's stroke pipeline runs without frame(); GUI editor becomes a thin front-end over the same commands.	An LLM paints rooms via the same bus a human uses.

---

11. Quick wins (afternoon-sized, near-zero risk — the recommended first move)

1. mrpgi --render-room rooms/room0.json out.png — bake a RoomDoc to PNG headless via the existing compose path. Proves headless rendering with almost no refactor; gives the LLM a way to preview authored rooms today.
2. Lift run_command() into mrpgi-core and unit-test it with golden (input → lines, understood) cases. Zero risk; instant safety net for the P1 refactor.
3. Generate the AI verb list in build_prompt (src/ai.rs:68) from canon_verb's table instead of a hardcoded string, so the two can never drift. ~15 min; also preps data-driven verbs.
4. Wrap sound calls behind trait AudioSink with the existing Sfx/Music as the impl (no behaviour change). Mechanical; unlocks the whole audio-options path later.

---

12. Risks & guardrails

• macroquad owns the window/audio/GL context via #[macroquad::main]; the core must never call into it. Enforce with the workspace boundary — a compile error is the guardrail. Watch assets.rs: the quantize()/Cel path is pure (image crate) but any macroquad Image/Texture use stays in the front-end.
• Determinism: the headless core must take dt explicitly and fix the cycle rate (CYCLE_DT = 1/20) so replays are bit-identical. Keep the (non-deterministic) AI lane out of the deterministic core path — it produces a Command that then runs through deterministic logic (the current design already respects this).
• Backward-compat: every new RoomDoc/ObjDef field MUST be #[serde(default)] (the codebase already does this) so the existing rooms/*.json and kit/ packs keep loading.
• Strudel is a genuine native/web split — accept a browser sidecar as the real path; don't over-invest in a native mini-notation interpreter until a native-only user needs it.
• Editor/Play share state — the core split is the chance to make World the single source of truth. Keep the GUI editor driving the same commands the server uses, rather than a parallel path, to avoid regressing the working editor.

---

13. Open questions / next step

Recommended opening play: P0 (carve mrpgi-core) + the four quick wins, on a branch — gets a headless engine rendering a room and passing its first tests without touching the fun stuff.

Or keep designing first: nail down the exact Command/Event enums and the game.json manifest schema before any code moves.

Still to decide when building starts:

• HTTP framework: axum vs. a minimal hyper/tiny_http (weigh against build times).
• MCP transport: stdio (simplest) to start.
• Whether FileSink lands in P2 (alongside game folders) or waits for P5.

---

14. Source map (current engine)

File	Lines	Role
src/main.rs	728	Window, game loop, mode switch, parser (run_command), compositing, HUD. The fusion to break up.
src/editor.rs	1362	In-engine paint tool; also the de-facto world database (RoomDoc, ObjDef, DlgNode, sprite catalog).
src/assets.rs	440	PNG → EGA quantize + dither; tile / pour-into-shape / object insertion; hot-reload.
src/picture.rs	283	Vector room painting (PicOp rects / lines / pixels), stamp_cel_fit.
src/sprite.rs	166	The ego: movement, click-to-walk, collision, walk animation; Prop.
src/ai.rs	134	AI parser lane (Ollama / OpenRouter / off) — the template for the whole option/adapter pattern.
src/sound.rs	151	Runtime square-wave chiptune synth + 5 music moods → WAV bytes.
src/framebuffer.rs	87	The visual + priority buffers and the band(y) depth table.
src/room.rs	72	Hand-authored demo Scene.
src/view.rs	51	Sprites as loops + cels, authored from ASCII grids.
src/palette.rs	40	The 16-color EGA palette (indices → RGBA).

Resources: rooms/*.json (room docs), kit/ (themed objects + locations — proto game-folders), sprites/*.png, assets/*.png, palettes/. Docs: README.md, MANUAL.md, EDITOR_GUIDE.md.

---

15. Provenance

This handover was produced from a full read of the engine source (src/*.rs) plus a parallel architecture-mapping pass over every subsystem (game loop, editor, render pipeline, actors, AI + sound, resources) and a synthesis design pass. It reflects the state of main @ d0caf98 (Initial commit) and the design conversation of 2026-07-05. Diagram source: architecture.svg.