# lessismore

Squeeze text before it hits an LLM. Deterministic passes first (free, safe,
reproducible); optional perplexity pruning via LLMLingua-2 for the last mile.

```bash
python3 lessismore.py dump.log -l 2 > small.log     # file in, file out
tail -5000 app.log | python3 lessismore.py -l 2      # pipe filter
python3 lessismore.py big.txt -l 2 --ml 0.5          # + LLMLingua (pip install llmlingua)
```

```python
from lessismore import compress, count_tokens
small = compress(big_log, level=2)
```

Token stats print to stderr, so pipes stay clean. Measured at level 2 with the
real o200k tokenizer (no ML, no deps): a 2000-line mixed error log went
**86,914 → 7,895 tokens (11x)**; an interleaved three-service log went
**55,999 → 4,587 (12.2x)**; an ANSI-colored CI build log went **21,198 → 2,387
(8.9x)**; captured pip/tqdm output with `\r` redraws compressed **98%**.

| Level | Passes | Safe for |
|---|---|---|
| 1 | whitespace collapse, duplicate-line runs | code structure & indentation (whitespace runs *inside string literals* still collapse) |
| 2 | + minify whole-input JSON, keep final state of `\r` progress redraws, strip ANSI + ISO timestamps, squash blobs/UUIDs/venv paths, alias scattered duplicate lines, collapse same-words-different-numbers runs | logs, dumps, tool output |
| 3 | + filler-word stripping | prose only (eats "just" everywhere) |
| 4 | + `two_sticks` caveman word-dropping | gist-only prose — never instructions (38% measured on chatty prose) |

`--ml RATE` adds LLMLingua-2 perplexity pruning after the deterministic passes
(RATE = fraction of tokens kept). Needs `pip install llmlingua`; `pip install
tiktoken` for exact token counts (falls back to chars/4).

## Design notes — where the wins actually are

1. **Compress the bulk, not the question.** A 30-token prompt isn't worth
   touching; the 50k-token log dump, retrieved doc, or file dump is. That's why
   this is a pipe filter, not a chat-prompt rewriter.

2. **Characters ≠ tokens.** Measured with o200k: `[fmt:md_tbl+hdr]` = 8 tokens
   vs `"Format the output as a markdown table with headers."` = 10.
   `[out:json_strict]` vs `"Respond only with valid JSON."` = 6 vs 6.
   Tokenizers already compress common English; bracket-shorthand gets shredded
   into fragments *and* adds ambiguity. Shorthand codebooks: skipped, measured, dead.

3. **Stop-word stripping: instructions no, gist yes.** Mangled grammar hurts
   instruction-following, so levels 1–3 never touch structure words. But every
   dropped word is a whole token, so for content you only need the gist of,
   level 4 (`two_sticks`) drops articles/copulas/auxiliaries — measured 38% on
   chatty prose. Negations, modals, and order words are never dropped: "do not
   delete" must stay "not delete", never "delete".

   The two-sticks *recoding* idea (map words to 2–3 letter codes) is dead on
   arrival, measured: common words are already 1 token (` database` = 1) while
   off-distribution codes cost 2 (` qx` = 2), plus a ~4-token/entry codebook
   tax. BPE already is a 200k-entry compression codebook; you can't beat it
   with an 18k-entry codebook written inside its own encoding. Zipping a zip.

   The salvage: codes DO pay when one code replaces a repeated *multi-token
   sequence* — that's `alias_repeats`, dictionary compression that composes
   with BPE instead of fighting it (12.2x measured on interleaved logs).

4. **Perplexity pruning is a dependency, not a project.** Microsoft's
   `llmlingua` already does the budget-model math. We wrap it in six lines
   behind `--ml` instead of reimplementing it. Deterministic passes run first
   so you're not paying a classifier to delete duplicate log lines.

5. **Determinism matters for prompt caching.** Same input → same output keeps
   cache prefixes stable. Corollary: never compress a stable cached system
   prompt — you'd bust the cache to save tokens that were already free.

6. **The compressor must cost less than it saves.** Regex passes are ~free at
   any size. The ML pass loads a model, so it only pays off on multi-KB inputs.

7. **The information-loss dial** from the original idea is the `-l` level +
   `--ml` rate: level 1 is lossless-ish and code-safe, `--ml 0.3` is maximum
   squeeze for prose you only need the gist of.

8. **Personal shorthand belongs on the keyboard, not in the model.** Your typed
   prompts are the cheapest tokens in the context (~10 each); a skill teaching
   the model your abbreviations costs more per turn than it can ever save.
   Expand shorthand client-side instead — Claude Code slash commands are the
   native mechanism, and `grunts.py` mines your own transcript history for
   what you actually retype (`--emit` writes the command stubs).

## Fleet setup (stupendo / m3 air)

```bash
git clone ssh://git@100.71.119.27:222/monster/lessismore.git ~/Documents/lessismore
printf '#!/bin/sh\nexec python3 ~/Documents/lessismore/lessismore.py "$@"\n' | sudo tee /opt/homebrew/bin/lm >/dev/null
sudo chmod +x /opt/homebrew/bin/lm
```

`lm` is the pipe shim: `docker logs dealgod | lm -l 2`. Optional: `pip3 install
tiktoken` for exact token stats (chars/4 heuristic otherwise).

## Known tradeoffs

- Levels 2+ assume line *order* matters but wall-clock timing doesn't. When
  gaps and deltas are the signal (hang hunting), stay on level 1.
- Output is for LLM consumption, not round-tripping: markdown hard breaks
  (trailing double-space) and diff context lines don't survive even level 1.
- At levels 3–4, dedupe counts describe the post-stripped text — five
  differently-phrased "retry the job" lines can legitimately merge.
- In-band markers can collide with input that already contains them;
  `alias_repeats` bails out if its own `@N` markers already appear as lines.