lessismore/README.md

lessismore 📉

Turn 50,000 tokens of log spam into 4,500 tokens of pure signal — before it ever hits your LLM.

lessismore is a deterministic compression filter for the bulk text that actually fills context windows: logs, CI output, tracebacks, JSON dumps, captured tool output. Pure stdlib Python. Zero dependencies. Every claim below was measured against the real o200k tokenizer — and every idea that failed the measurement is documented at the bottom, so you know exactly what you're getting.

tail -5000 app.log | python3 lessismore.py | llm "why did this crash?"

11x on mixed error logs · 16x on ANSI CI logs · 97% on captured pip/tqdm output · 3.0x on real LogHub system logs · 95% critical-signal retention on the LogDx-CI benchmark's 35 real GitHub Actions failures · ~6–12 MB/s single-core · 0 dependencies

And the part a compression ratio can't prove: in the root-cause eval (eval.py), the model found the planted root cause in 8/8 compressed logs vs 6/8 raw — smaller and more often right. On 12 real LogDx-CI failures it scored 12/12 on both raw and compressed, at less than half the tokens.

Why this exists

Three facts about LLM context, and the gap between them is this tool:

1. The expensive part isn't your prompt. Your typed question is ~30 tokens. The log dump you attach is 50,000. Compress the wall, not the question.
2. Machine output is mostly redundancy. ANSI color codes, \r progress redraws, timestamps, venv path spam, the same error line 400 times — noise that looks small on a terminal screen but is real tokens in a capture.
3. BPE tokenizers already compress English (common words are 1 token — you can't out-abbreviate them, we checked). What BPE can't see is repetition across a file or terminal noise. That's the seam this tool mines.

Three payoffs when you pipe through it:

• Context capacity — an hour of log history fits where two minutes did. On a local model, that's the difference between full speed and crawling.
• Prompt-cache longevity — every pass is deterministic and idempotent: same input, same output, byte for byte (it's a test invariant). This matters wherever a prompt is re-generated from source each call — a CI assistant re-reading the same log, RAG context rebuilt per request, a hook re-running on every tool call. A deterministic pass produces the identical prefix every time, so the provider cache hits; an ML compressor in the loop busts it on every subtle variation. (Within a single chat, follow-ups hit the cache regardless — the transcript is append-only. The claim is about rebuilt prompts, not chat turns.)
• Model attention — LLMs lose things in the middle of walls of text. Feed signal, not noise, and the first answer is the right one more often.

Quickstart

pip install .                # from a clone — installs the `lessismore` and `lm` commands
pip install '.[tokens]'      # + tiktoken for exact token stats (else chars/4 estimate)
python3 test_lessismore.py   # should print "ok"

lm dump.log > small.log                 # file in, file out (level 2 is the default)
docker logs myapp 2>&1 | lm             # pipe filter
lm --run "pytest -x"                    # run a noisy command, print it compressed,
                                        #   keep its exit code (pipes can't)
lm big.txt --budget 4000                # compress, then hard-cap at ~4k tokens
lm big.txt --ml 0.5                     # + LLMLingua last mile
lm --serve                              # paste-in demo page on localhost:7777

No install needed either — python3 lessismore.py works the same from a bare clone; it's one stdlib-only file.

from lessismore import compress, count_tokens, budget
small = budget(compress(big_log, level=2), 8000)

Token stats print to stderr, so pipes stay clean.

The dial

Four levels, from byte-cautious to caveman. Pick by content, not by greed.

Level	What it eats	Point it at	Profile
1	whitespace runs, consecutive duplicate lines	code, scripts, anything	structure-safe: indentation and content untouched (whitespace inside string literals still collapses)
2	+ JSON minify (whole-input and embedded pretty blocks), \r redraw collapse, ANSI strip, timestamps (ISO/syslog/nginx/HDFS/BGL/HealthApp), base64/base64url/hex blobs, UUIDs, venv paths, scattered-duplicate aliasing, template mining, similar-line collapse with value summaries	logs, CI dumps, traces, tool output	the sweet spot and the default — destroys machine noise, keeps every distinct fact
3	+ filler-phrase stripping ("could you please", "just")	prose, chat history	fine for text, never for strict logic
4	+ two_sticks caveman mode: drops articles/copulas/auxiliaries, never negations or modals	gist-only prose, transcripts	lossy on style, protective of meaning — "do not delete" keeps its not

"Keeps every distinct fact" is enforced, not hoped: when similar lines collapse, the digits that varied are summarized in the marker — [3 similar lines omitted; values 404/429/503] — because sometimes the digits (status codes, ports, exit codes) are the diagnosis. An earlier version silently ate them; the adversarial eval below is what caught it.

The crown jewels at level 2 are the two dictionary passes:

• alias_repeats — ordinary dedupe only sees consecutive repeats, so interleaved multi-service logs sail straight through it. This pass hunts scattered exact duplicates across the whole file and dictionary-codes them (@1 = ERROR [pool-3] psycopg2... once in a legend, 2-token @1 everywhere else). Lossless — the legend keeps every line verbatim. On the interleaved benchmark: 54,999 → 34,999 with dedupe alone (1.6x) vs 54,999 → 4,585 with aliasing (12x).
• alias_templates — real logs rarely repeat exact lines; they repeat templates: Failed password for root from 10.2.3.4 port 2201. Lines that differ only in their digit-bearing tokens get one legend entry (@t1 = Failed password for root from <*> port <*> ssh2) and per-line values (@t1 10.2.3.4 2201). Digit-bearing tokens that never vary (ssh2, v1.2.3) are inlined into the template. This is a deterministic, single-pass Drain-lite, and it's what benchmarking on real logs demanded: it took the LogHub corpus from 1.8x to 3.0x.

--budget N is the backstop, not the compressor: after the passes run, it keeps head and tail lines and drops the middle with a [~N tokens omitted] marker. It's the only pass with guaranteed bounded output — use it when the context limit is a hard wall.

The receipts

Every row reproducible from one command: python3 bench.py (o200k counts via tiktoken — it refuses to print estimate numbers, so what you measure is what this table claims).

Sample	Tokens	Why it wins
Mixed error log, 2000 lines	86,914 → 7,619 (11.4x)	timestamp strip unmasks identical lines → dedupe; templates catch the numbered stragglers
Interleaved 3-service log, zero consecutive repeats	54,999 → 4,585 (12.0x)	alias_repeats — consecutive dedupe alone managed 1.6x on this input
ANSI-colored CI log	10,440 → 657 (15.9x)	color codes make identical lines look different; strip them and the log collapses
Captured pip/tqdm output	10,465 → 265 (97%)	every overwritten \r progress frame is invisible on screen but real tokens in a capture
pytest failure dump	732 → 515 (30%)	one site-packages path prefix = 25 tokens → 7
Pretty-printed JSON API response	8,932 → 3,234 (64%)	minify (round-trip verified lossless) + UUID→8-hex squash
Chatty prose, level 4	427 → 264 (38%)	every dropped function word is a whole token; level 3 got 2% on the same text

Throughput: measured 6–12 MB/s single-core at level 2 (20 MB log in 2.6s). No model, no network — break-even input size is effectively zero.

Real logs, real benchmarks

Synthetic logs flatter a compressor — they repeat exactly the way its passes expect. So bench_real.py runs three public corpora with deterministic metrics (no model in the loop, no cherry-picking):

LogHub samples — 10 real system logs (HDFS, BGL, OpenSSH, OpenStack...), 1.15M tokens: 3.0x overall, ranging from 1.5x (HealthApp's pipe-format) to 6.5x (Apache). Fair warning made explicit: real diverse logs compress 2–4x, not the 11x of repetitive error walls. It was this benchmark that forced alias_templates and four new timestamp formats into existence — the corpus started at 1.5x.

LogChunks — 797 real Travis CI failure logs where humans marked the chunk that explains each failure (externally validated with the developers who caused them). 66.2M tokens → 34.2M (1.9x), and 78.5% of labeled failure-explaining lines survive byte-for-byte; 73% of chunks survive fully intact. The rest is dominated by similar-line collapse where the message survives once with a value summary — represented, not deleted.

LogDx-CI — the benchmark built for exactly this question: 35 real GitHub Actions failures with per-case ground-truth diagnostic signals. All 35 cases, 15.0M tokens → 6.9M (2.2x), with 95.2% of critical diagnostic signals retained (139/146; of the 7 misses, most are layout artifacts where templating separates a value from its line). With --budget 8000 stacked on top: 6,334 tokens/case at 77.4% critical-signal retention — the same token cost as the benchmark's tail-200 baseline, with the whole log's worth of scattered signals instead of just the tail. And the answer-quality check: on 12 cases, claude-haiku-4.5 diagnosed 12/12 from raw and 12/12 from compressed — parity at ~2.4x fewer tokens. (Ground-truth caveat inherited from the benchmark: its labels are AI-drafted with single-author verification, n=35.)

One case in this run earned its keep twice: the benchmark caught a real lessismore bug (a camelCase test name eaten as a base64 blob), which is now a regression test. That's what public benchmarks are for.

Does the model still get the answer?

A compression ratio proves the log got smaller, not that the signal survived. So this is the eval that matters, and it's in the repo: eval.py generates eight seeded failure logs, each with one planted root cause buried in machine noise plus a red herring (e.g. a cascade of connection errors caused by an OOM kill 400 lines earlier), asks a model "what's the root cause?" on the raw and the compressed version, and grades the answer with a deterministic keyword check.

Result (claude-haiku-4.5, 2026-07-07, one run; this run went through the Claude Code agent harness with the same single-turn prompts — eval.py's default runner is a logged-in claude -p. Rerun it yourself):

Scenario	Raw tokens	Compressed	Raw	Compressed
OOM kill buried mid-log, connection-error red herring	38,777	174 (223x)	✗ blamed the red herring	✓ found the OOM
429 rate-limit burst hidden in a wall of status codes	46,899	3,713	✗ "service overwhelmed"	✓ named the 429s
Disk-full scattered through 3 interleaved services	38,616	8,103	✓	✓
One failed assertion in 1,400 ANSI test lines	46,812	21,563	✓	✓
Dependency conflict under progress-bar walls	16,304	355	✓	✓
Expired TLS cert in an nginx access-log wall	56,696	189 (300x)	✓	✓
Missing env var in a crash-loop traceback	7,680	1,610	✓	✓
Java deadlock in a 40-thread dump	8,157	7,907 (1.0x)	✓	✓
Total	259,941	43,614 (6.0x)	6/8	8/8

The two raw failures are the lost-in-the-middle effect this tool exists to counter: the model latched onto the loud symptom cascade and never surfaced the quiet cause. Compressed, the cause is impossible to miss. The honest caveats: n=1 run per condition, synthetic logs authored by this project, one model — which is exactly why the harness ships in the repo with fixed seeds. python3 eval.py --dry costs nothing and shows the scenarios; point LM_EVAL_CMD at any prompt-on-stdin CLI to grade your own model.

Where the tokens actually are: agent harnesses

A human remembering to pipe is the demo. The 50k-token walls of 2026 enter context windows as tool output inside agent harnesses, so lessismore ships two integrations:

lm --run — run the noisy command through the compressor and keep its exit code (a shell pipe can't do that without pipefail games):

lm --run "pytest -x"          # agent sees 500 tokens, not 20,000
lm --run "docker build ."

Tell your agent about it once (CLAUDE.md: "run noisy commands through lm --run") and every test run gets cheaper.

lm --hook — a Claude Code PostToolUse hook that compresses Bash output before it enters the model's context. Zero dependencies, no jq; outputs under 2,000 chars pass through untouched. In .claude/settings.json:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [{ "type": "command", "command": "lm --hook" }]
      }
    ]
  }
}

Because the passes are deterministic, hook-compressed transcripts stay byte-stable across re-runs — the prompt-cache argument above is strongest exactly here.

What we refused to build (measured so it stays dead)

The graveyard is a feature. Each of these looks clever on paper and loses against a real tokenizer:

• Shorthand codebooks — [fmt:md_tbl+hdr] costs 8 tokens; "Format the output as a markdown table with headers." costs 10. BPE already has English baked in; bracket syntax shreds into off-distribution fragments.
• Word→code recoding ("database" → qx) — common words are already 1 token, random codes cost 2, plus ~4 tokens/entry of codebook tax. You cannot beat a 200k-entry codebook from inside its own encoding. Zipping a zip.
• Known abbreviations (db, fn, env, auth) — measured 1 → 1 tokens. Zero.
• Personalized shorthand skills — scanned 2,917 real typed prompts across 654 transcripts: 2,762 unique, and the repeats were already grunts ("yes", "go"). A model-side decode skill costs ~1k tokens/turn to save ~10. grunts.py keeps the half that works: mine your own history, emit client-side slash-command stubs — expansion before the model sees it is free.
• Digit-masked dedupe (0.0% — progress bars differ in glyphs, not digits), separator shortening (a 78-char ---- is already 1 token), prefix hoisting (one stray line kills it), JSON→TSV / float truncation / pointer squash (real savings, worse trade).

Battle-tested

An adversarial review agent was told to break it and confirmed 14 real failure modes — negation-inverting filler stripping ("was not just the db" → "was not the db"), URLs eaten as base64, crashes on empty and non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated", two_sticks eating "IT" and "US" as function words. Nine fixed with regression tests, four documented below, one wontfix (adversarial in-band marker collision). A second review round found the big one: similar-line collapse silently eating distinct status codes — fixed with value summaries and now guarded by the downstream eval. python3 test_lessismore.py — 42 asserts, no framework.

Where it does nothing (on purpose)

Savings are proportional to redundancy, not size. This removes repetition and machine noise; it cannot compress information, and doesn't pretend to.

Content	Expect	Verdict
Repetitive machine output	5–16x, up to 300x on pathological repeats	the reason this exists
Real-world CI / system logs	2–4x (measured on LogHub, LogChunks, LogDx-CI)	the honest production number
Structured data (JSON, tracebacks)	30–64%	worthwhile, lossless where it fires
Varied prose	~2% (level 3) / 38% lossy (level 4)	gist mode only
Clean code, unique dense text, thread dumps	~0% by design	that's what --ml or truncation is for

--ml RATE bolts on Microsoft's LLMLingua-2 for perplexity pruning as a last mile (pip install llmlingua) — runs after the deterministic passes so you're not paying a classifier model to delete duplicate log lines. Only worth it on multi-KB inputs, and it forfeits the cache-stability guarantee.

Related work (know the river you're panning)

Honest placement, so you can pick the right tool:

• rtk — Rust proxy with per-command filters (ANSI strip, dedupe, failures-only test output), hooks into agent CLIs. Command-aware and aggressive where lessismore is generic text passes and lossless-leaning: rtk decides what you need to see; lessismore removes only what is provably redundant and keeps a legend.
• Headroom — the heavyweight tool-output compressor (Python, proxy + MCP). Has ML models in the loop, so it needs a cache-alignment component to patch the non-determinism lessismore doesn't have.
• Drain3 — industrial log template mining. alias_templates is a deterministic, single-pass, zero-dep Drain-lite (digit-token masking instead of similarity clustering); if you need streaming template state across files or letter-varying parameters (usernames, hostnames), use Drain3.
• LLMLingua-2 — ML token pruning; wrapped here as the optional --ml last mile, after the deterministic passes, so you never pay a classifier to delete duplicate log lines.
• Dictionary-encoding prompt compression (2026) — independently validates the @1 = line legend technique academically (≥0.99 fidelity on log benchmarks); no code released. lessismore is, in effect, a reference implementation.

What none of them package together — and the reason this exists — is the combination: pure-stdlib, deterministic end to end, tokenizer-measured, with the cache-stability argument as a design constraint rather than a patch.

Try it in a browser

lm --serve runs a paste-in demo page on http://localhost:7777 — paste your ugliest log, pick a level, watch the token count drop. Stdlib only, binds localhost only.

License

MIT — see LICENSE.

Known tradeoffs

• Levels 2+ assume line order matters but wall-clock timing doesn't. When gaps and deltas are the signal (hang hunting), stay on level 1.
• Output is for LLM consumption, not round-tripping: markdown hard breaks (trailing double-space) and diff context lines don't survive even level 1.
• At levels 3–4, dedupe counts describe the post-stripped text — five differently-phrased "retry the job" lines can legitimately merge.
• In-band markers can collide with input that already contains them; alias_repeats bails out if its own @N markers already appear as lines.
• Blob squashing eats base64url runs (real JWTs) only when they look like blobs (digits + mixed case); a 64-char kebab-case or snake_case identifier is treated as content and survives.
• Similar-line collapse summarizes varying digits per column (values 404/429/503, or a min–max range past 4 distinct); if the digit runs per line differ in count, it falls back to a plain count marker.
• alias_templates reconstructs lines by filling <*> left to right with the per-line values — lossless modulo whitespace runs (a templated line's original spacing/alignment is not preserved). Parameters that vary in letters (usernames, hostnames) split into separate templates rather than merging — less compression, never confusion.