lessismore/README.md

# lessismore 📉

**Turn 50,000 tokens of log spam into 4,500 tokens of pure signal — before it
ever hits your LLM.**

lessismore is a deterministic compression filter for the bulk text that
actually fills context windows: logs, CI output, tracebacks, JSON dumps,
captured tool output. Pure stdlib Python. Zero dependencies. Every claim below
was measured against the real o200k tokenizer — and every idea that failed the
measurement is documented at the bottom, so you know exactly what you're
getting.

```bash
tail -5000 app.log | python3 lessismore.py | llm "why did this crash?"
```

**13x** on mixed error logs · **16x** on ANSI CI logs · **98%** on captured
pip/tqdm output · **3.9x** on real LogHub system logs · **95% critical-signal
retention** on the LogDx-CI benchmark's 35 real GitHub Actions failures — and
at an equal 2k-token budget, compress-then-truncate keeps **1.9x more**
diagnostic signals than plain truncation · **~6–12 MB/s** single-core ·
**0** dependencies

And the part a compression ratio can't prove: in the root-cause eval
([eval.py](eval.py)), the model found the planted root cause in **8/8**
compressed logs vs **6/8** raw — smaller *and* more often right. On 12 real
LogDx-CI failures it scored 12/12 on both raw and compressed, at less than
half the tokens.

## Why this exists

Three facts about LLM context, and the gap between them is this tool:

1. **The expensive part isn't your prompt.** Your typed question is ~30
   tokens. The log dump you attach is 50,000. Compress the wall, not the
   question.
2. **Machine output is mostly redundancy.** ANSI color codes, `\r` progress
   redraws, timestamps, venv path spam, the same error line 400 times — noise
   that looks small on a terminal screen but is real tokens in a capture.
3. **BPE tokenizers already compress English** (common words are 1 token —
   you can't out-abbreviate them, we checked). What BPE *can't* see is
   repetition across a file or terminal noise. That's the seam this tool
   mines.

Three payoffs when you pipe through it:

- **Context capacity** — an hour of log history fits where two minutes did.
  On a local model, that's the difference between full speed and crawling.
- **Prompt-cache longevity** — every pass is deterministic and idempotent:
  same input, same output, byte for byte (it's a test invariant). This
  matters wherever a prompt is *re-generated* from source each call — a CI
  assistant re-reading the same log, RAG context rebuilt per request, a hook
  re-running on every tool call. A deterministic pass produces the identical
  prefix every time, so the provider cache hits; an ML compressor in the
  loop busts it on every subtle variation. (Within a single chat, follow-ups
  hit the cache regardless — the transcript is append-only. The claim is
  about rebuilt prompts, not chat turns.)
- **Model attention** — LLMs lose things in the middle of walls of text. Feed
  signal, not noise, and the first answer is the right one more often.

## Quickstart

```bash
pip install .                # from a clone — installs the `lessismore` and `lm` commands
pip install '.[tokens]'      # + tiktoken for exact token stats (else chars/4 estimate)
python3 test_lessismore.py   # should print "ok"
```

```bash
lm dump.log > small.log                 # file in, file out (level 2 is the default)
docker logs myapp 2>&1 | lm             # pipe filter
lm --run "pytest -x"                    # run a noisy command, print it compressed,
                                        #   keep its exit code (pipes can't)
lm big.txt --budget 4000                # compress, then hard-cap at ~4k tokens
lm big.txt --ml 0.5                     # + LLMLingua last mile
lm --serve                              # paste-in demo page on localhost:7777
```

No install needed either — `python3 lessismore.py` works the same from a bare
clone; it's one stdlib-only file.

```python
from lessismore import compress, count_tokens, budget
small = budget(compress(big_log, level=2), 8000)
```

Token stats print to stderr, so pipes stay clean.

## The dial

Four levels, from byte-cautious to caveman. Pick by content, not by greed.

| Level | What it eats | Point it at | Profile |
|---|---|---|---|
| **1** | whitespace runs, consecutive duplicate lines | code, scripts, anything | structure-safe: indentation and content untouched (whitespace *inside string literals* still collapses) |
| **2** | + JSON minify (whole-input *and* embedded pretty blocks), `\r` redraw collapse, ANSI strip, timestamps (ISO/syslog/nginx/HDFS/BGL/HealthApp), base64/base64url/hex blobs, UUIDs, venv paths, scattered-duplicate aliasing, template mining, similar-line collapse with value summaries | logs, CI dumps, traces, tool output | **the sweet spot and the default** — destroys machine noise, keeps every distinct fact |
| **3** | + filler-phrase stripping ("could you please", "just") | prose, chat history | fine for text, never for strict logic |
| **4** | + `two_sticks` caveman mode: drops articles/copulas/auxiliaries, never negations or modals | gist-only prose, transcripts | lossy on style, protective of meaning — "do **not** delete" keeps its *not* |

"Keeps every distinct fact" is enforced, not hoped: when similar lines
collapse, the digits that varied are summarized in the marker —
`[3 similar lines omitted; values 404/429/503]` — because sometimes the
digits (status codes, ports, exit codes) *are* the diagnosis. An earlier
version silently ate them; the adversarial eval below is what caught it.

The crown jewels at level 2 are the two dictionary passes:

- `alias_repeats` — ordinary dedupe only sees *consecutive* repeats, so
  interleaved multi-service logs sail straight through it. This pass hunts
  scattered exact duplicates across the whole file and dictionary-codes them
  (`@1 = ERROR [pool-3] psycopg2...` once in a legend, 2-token `@1`
  everywhere else). Lossless — the legend keeps every line verbatim. On the
  interleaved benchmark: 54,999 → 34,999 with dedupe alone (1.6x) vs
  54,999 → 4,585 with aliasing (**12x**).
- `alias_templates` — real logs rarely repeat *exact* lines; they repeat
  templates: `Failed password for root from 10.2.3.4 port 2201`. Two tiers of
  deterministic, single-pass Drain-lite mining. Tier A masks hex/digit runs
  *inside* tokens, so run-level constants inline into the template
  (`@t1 = Failed password for root from 10.2.3.<*> port <*> ssh2`, per-line
  refs `@t1 44 2201`). Tier B masks whole digit-bearing tokens on whatever
  tier A left behind, so paths and ids that vary in letters still group —
  which is what crushes compile/build logs. Each group is costed with exact
  character accounting under both value encodings and takes the cheaper; a
  template only fires when it provably pays. Every tier exists because a
  public benchmark regressed without it: this pass took the LogHub corpus
  from 1.8x to 3.9x.

`--budget N` is the backstop, not the compressor: after the passes run, it
keeps head and tail lines and drops the middle with a `[~N tokens omitted]`
marker. It's the only pass with *guaranteed* bounded output — use it when the
context limit is a hard wall.

## The receipts

Every row reproducible from one command: `python3 bench.py` (o200k counts via
tiktoken — it refuses to print estimate numbers, so what you measure is what
this table claims).

| Sample | Tokens | Why it wins |
|---|---|---|
| Mixed error log, 2000 lines | 86,914 → 6,730 (**12.9x**) | timestamp strip unmasks identical lines → dedupe; templates catch the numbered stragglers |
| Interleaved 3-service log, zero consecutive repeats | 54,999 → 4,585 (**12.0x**) | `alias_repeats` — consecutive dedupe alone managed 1.6x on this input |
| ANSI-colored CI log | 10,440 → 668 (**15.6x**) | color codes make identical lines look different; strip them and the log collapses |
| Captured pip/tqdm output | 10,465 → 216 (**98%**) | every overwritten `\r` progress frame is invisible on screen but real tokens in a capture |
| pytest failure dump | 732 → 515 (30%) | one site-packages path prefix = 25 tokens → 7 |
| Pretty-printed JSON API response | 8,932 → 3,234 (**64%**) | minify (round-trip verified lossless) + UUID→8-hex squash |
| Chatty prose, level 4 | 427 → 264 (38%) | every dropped function word is a whole token; level 3 got 2% on the same text |

Throughput: measured **6–12 MB/s single-core** at level 2 (20 MB log in
2.6s). No model, no network — break-even input size is effectively zero.

## Real logs, real benchmarks

Synthetic logs flatter a compressor — they repeat exactly the way its passes
expect. So [bench_real.py](bench_real.py) runs three **public** corpora with
deterministic metrics (no model in the loop, no cherry-picking):

**[LogHub](https://github.com/logpai/loghub) samples** — 10 real system logs
(HDFS, BGL, OpenSSH, OpenStack...), 1.15M tokens: **3.9x overall**, ranging
from 2.0x (HDFS's unique block ids) to 9.8x (OpenStack). Fair warning made
explicit: real diverse logs compress 2–10x by structure, not the flat 13x of
repetitive error walls. It was this benchmark that forced `alias_templates`
and four new timestamp formats into existence — the corpus started at 1.5x.

**[LogChunks](https://zenodo.org/records/3632351)** — 797 real Travis CI
failure logs where *humans* marked the chunk that explains each failure
(externally validated with the developers who caused them). 66.2M tokens →
31.6M (**2.1x**), and **76.9% of labeled failure-explaining lines survive
byte-for-byte**; 67.5% of chunks survive fully intact. The rest is dominated
by similar-line collapse and templating, where the message survives once with
per-line values — represented, not deleted.

**[LogDx-CI](https://arxiv.org/abs/2605.28876)** — the benchmark built for
exactly this question: 35 real GitHub Actions failures with per-case
ground-truth diagnostic signals. All 35 cases, 15.0M tokens → 6.6M
(**2.3x**), with **95.2% of critical diagnostic signals retained** (139/146;
of the 7 misses, most are layout artifacts where templating separates a value
from its line). And the answer-quality check: on 12 cases, claude-haiku-4.5
diagnosed **12/12 from raw and 12/12 from compressed** — parity at ~2.4x
fewer tokens. (Ground-truth caveat inherited from the benchmark: its labels
are AI-drafted with single-author verification, n=35.)

**The equal-budget receipt** — the question every harness actually faces is
not "compress or not" but "we cap tool output anyway; is compressing before
the cap worth it?" Measured on all 35 LogDx-CI cases (critical-signal
retention at the same token budget):

| Budget | truncate only | compress, then truncate |
|---|---|---|
| 2,000 tokens | 24.7% | **46.6%** |
| 8,000 tokens | 65.8% | **78.8%** |
| 32,000 tokens | 80.8% | **87.7%** |

One case in this run earned its keep twice: the benchmark caught a real
lessismore bug (a camelCase test name eaten as a base64 blob), which is now a
regression test. That's what public benchmarks are for.

A faithfulness note that cost real ratio: an earlier build scored "8.9x" on
LogHub because refs from *different* templates were collapsing into one
"similar lines" blob, silently dropping the middle lines' values. That
over-merge is now prevented (a ref's template name is part of its identity),
and the honest number is 3.9x. The same audit exposed that v0.3.0's HDFS
"win" was powered by the same phantom — this codebase prefers the smaller
true number.

## Does the model still get the answer?

A compression ratio proves the log got smaller, not that the signal
survived. So this is the eval that matters, and it's in the repo:
[eval.py](eval.py) generates eight seeded failure logs, each with one planted
root cause buried in machine noise **plus a red herring** (e.g. a cascade of
connection errors *caused by* an OOM kill 400 lines earlier), asks a model
"what's the root cause?" on the raw and the compressed version, and grades
the answer with a deterministic keyword check.

Result (claude-haiku-4.5, 2026-07-07, one run; this run went through the
Claude Code agent harness with the same single-turn prompts — `eval.py`'s
default runner is a logged-in `claude -p`. Rerun it yourself):

| Scenario | Raw tokens | Compressed | Raw | Compressed |
|---|---|---|---|---|
| OOM kill buried mid-log, connection-error red herring | 38,777 | 169 (**229x**) | ✗ blamed the red herring | ✓ found the OOM |
| 429 rate-limit burst hidden in a wall of status codes | 46,899 | 3,497 | ✗ "service overwhelmed" | ✓ named the 429s |
| Disk-full scattered through 3 interleaved services | 38,616 | 8,099 | ✓ | ✓ |
| One failed assertion in 1,400 ANSI test lines | 46,812 | 11,408 | ✓ | ✓ |
| Dependency conflict under progress-bar walls | 16,304 | 355 | ✓ | ✓ |
| Expired TLS cert in an nginx access-log wall | 56,696 | 11,855 | ✓ | ✓ |
| Missing env var in a crash-loop traceback | 7,680 | 1,610 | ✓ | ✓ |
| Java deadlock in a 40-thread dump | 8,157 | 3,742 (2.2x) | ✓ | ✓ |
| **Total** | **259,941** | **40,735 (6.4x)** | **6/8** | **8/8** |

The two raw failures are the lost-in-the-middle effect this tool exists to
counter: the model latched onto the loud symptom cascade and never surfaced
the quiet cause. Compressed, the cause is impossible to miss. The honest
caveats: n=1 run per condition, synthetic logs authored by this project, one
model — which is exactly why the harness ships in the repo with fixed seeds.
`python3 eval.py --dry` costs nothing and shows the scenarios; point
`LM_EVAL_CMD` at any prompt-on-stdin CLI to grade your own model.

## Where the tokens actually are: agent harnesses

A human remembering to pipe is the demo. The 50k-token walls of 2026 enter
context windows as **tool output inside agent harnesses**, so lessismore
ships two integrations:

**`lm --run`** — run the noisy command through the compressor and keep its
exit code (a shell pipe can't do that without pipefail games):

```bash
lm --run "pytest -x"          # agent sees 500 tokens, not 20,000
lm --run "docker build ."
```

Tell your agent about it once (CLAUDE.md: *"run noisy commands through
`lm --run`"*) and every test run gets cheaper.

**`lm --hook`** — a Claude Code `PostToolUse` hook that compresses Bash
output before it enters the model's context. Zero dependencies, no `jq`;
outputs under 2,000 chars pass through untouched. In
`.claude/settings.json`:

```json
{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [{ "type": "command", "command": "lm --hook" }]
      }
    ]
  }
}
```

Because the passes are deterministic, hook-compressed transcripts stay
byte-stable across re-runs — the prompt-cache argument above is strongest
exactly here.

## What we refused to build (measured so it stays dead)

The graveyard is a feature. Each of these looks clever on paper and loses
against a real tokenizer:

- **Shorthand codebooks** — `[fmt:md_tbl+hdr]` costs **8 tokens**; "Format
  the output as a markdown table with headers." costs **10**. BPE already has
  English baked in; bracket syntax shreds into off-distribution fragments.
- **Word→code recoding** ("database" → `qx`) — common words are already 1
  token, random codes cost 2, plus ~4 tokens/entry of codebook tax. You
  cannot beat a 200k-entry codebook from inside its own encoding. Zipping a
  zip.
- **Known abbreviations** (db, fn, env, auth) — measured 1 → 1 tokens. Zero.
- **Personalized shorthand skills** — scanned 2,917 real typed prompts across
  654 transcripts: 2,762 unique, and the repeats were already grunts ("yes",
  "go"). A model-side decode skill costs ~1k tokens/turn to save ~10.
  `grunts.py` keeps the half that works: mine your own history, emit
  *client-side* slash-command stubs — expansion before the model sees it is
  free.
- **Digit-masked dedupe** (0.0% — progress bars differ in glyphs, not
  digits), **separator shortening** (a 78-char `----` is already 1 token),
  **prefix hoisting** (one stray line kills it), **JSON→TSV / float
  truncation / pointer squash** (real savings, worse trade).
- **Drain-style letter-varying template merging** (clustering templates that
  differ in ≤N word positions, so usernames/hostnames join one template) —
  measured on LogHub: 3.0x → 3.0x, with Linux *regressing* 6.4x → 5.3x. The
  extra per-line wildcard values eat exactly what the merged legends save.
- **Cross-template ref collapse** — letting `dedupe_similar` merge refs from
  different templates scored 8.9x on LogHub, but the "compression" was
  silently deleting the middle lines' values. Removed on faithfulness
  grounds; the ratio was a lie.

## Battle-tested

An adversarial review agent was told to break it and confirmed 14 real
failure modes — negation-inverting filler stripping ("was **not just** the
db" → "was **not** the db"), URLs eaten as base64, crashes on empty and
non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated",
`two_sticks` eating "IT" and "US" as function words. Nine fixed with
regression tests, four documented below, one wontfix (adversarial in-band
marker collision). A second review round found the big one: similar-line
collapse silently eating distinct status codes — fixed with value summaries
and now guarded by the downstream eval. `python3 test_lessismore.py` —
42 asserts, no framework.

## Where it does nothing (on purpose)

Savings are proportional to **redundancy, not size**. This removes repetition
and machine noise; it cannot compress information, and doesn't pretend to.

| Content | Expect | Verdict |
|---|---|---|
| Repetitive machine output | 5–16x, up to 300x on pathological repeats | the reason this exists |
| Real-world CI / system logs | 2–4x typical, up to 10x on template-heavy logs (measured on LogHub, LogChunks, LogDx-CI) | the honest production number |
| Structured data (JSON, tracebacks) | 30–64% | worthwhile, lossless where it fires |
| Varied prose | ~2% (level 3) / 38% lossy (level 4) | gist mode only |
| Clean code, unique dense text, thread dumps | ~0% by design | that's what `--ml` or truncation is for |

`--ml RATE` bolts on Microsoft's LLMLingua-2 for perplexity pruning as a
last mile (`pip install llmlingua`) — runs *after* the deterministic passes so
you're not paying a classifier model to delete duplicate log lines. Only
worth it on multi-KB inputs, and it forfeits the cache-stability guarantee.

## Related work (know the river you're panning)

Honest placement, so you can pick the right tool:

- **[rtk](https://github.com/rtk-ai/rtk)** — Rust proxy with per-command
  filters (ANSI strip, dedupe, failures-only test output), hooks into agent
  CLIs. Command-aware and aggressive where lessismore is generic text passes
  and lossless-leaning: rtk decides what you need to see; lessismore removes
  only what is provably redundant and keeps a legend.
- **[Headroom](https://github.com/chopratejas/headroom)** — the heavyweight
  tool-output compressor (Python, proxy + MCP). Has ML models in the loop,
  so it needs a cache-alignment component to patch the non-determinism
  lessismore doesn't have.
- **[Drain3](https://github.com/logpai/Drain3)** — industrial log template
  mining. `alias_templates` is a deterministic, single-pass, zero-dep
  Drain-lite (digit-token masking instead of similarity clustering); if you
  need streaming template state across files or letter-varying parameters
  (usernames, hostnames), use Drain3.
- **[LLMLingua-2](https://github.com/microsoft/LLMLingua)** — ML token
  pruning; wrapped here as the optional `--ml` last mile, *after* the
  deterministic passes, so you never pay a classifier to delete duplicate
  log lines.
- **[Dictionary-encoding prompt compression](https://arxiv.org/abs/2604.13066)**
  (2026) — independently validates the `@1 = line` legend technique
  academically (≥0.99 fidelity on log benchmarks); no code released.
  lessismore is, in effect, a reference implementation.

What none of them package together — and the reason this exists — is the
combination: pure-stdlib, deterministic end to end, tokenizer-measured, with
the cache-stability argument as a design constraint rather than a patch.

## Try it in a browser

`lm --serve` runs a paste-in demo page on `http://localhost:7777` — paste your
ugliest log, pick a level, watch the token count drop. Stdlib only, binds
localhost only.

## License

MIT — see [LICENSE](LICENSE).

## Known tradeoffs

- Levels 2+ assume line *order* matters but wall-clock timing doesn't. When
  gaps and deltas are the signal (hang hunting), stay on level 1.
- Output is for LLM consumption, not round-tripping: markdown hard breaks
  (trailing double-space) and diff context lines don't survive even level 1.
- At levels 3–4, dedupe counts describe the post-stripped text — five
  differently-phrased "retry the job" lines can legitimately merge.
- In-band markers can collide with input that already contains them;
  `alias_repeats` bails out if its own `@N` markers already appear as lines.
- Blob squashing eats base64url runs (real JWTs) only when they look like
  blobs (digits + mixed case); a 64-char kebab-case or snake_case identifier
  is treated as content and survives.
- Similar-line collapse summarizes varying digits per column
  (`values 404/429/503`, or a min–max range past 4 distinct); if the digit
  *runs* per line differ in count, it falls back to a plain count marker.
- `alias_templates` reconstructs lines by filling `<*>` left to right with
  the per-line values — lossless modulo whitespace runs (a templated line's
  original spacing/alignment is not preserved). Parameters that vary in
  *letters* (usernames, hostnames) split into separate templates rather than
  merging — less compression, never confusion.