lessismore/README.md

lessismore 📉

Turn 50,000 tokens of log spam into 4,500 tokens of pure signal — before it ever hits your LLM.

lessismore is a deterministic compression filter for the bulk text that actually fills context windows: logs, CI output, tracebacks, JSON dumps, captured tool output. Pure stdlib Python. Zero dependencies. Every claim below was measured against the real o200k tokenizer — and every idea that failed the measurement is documented at the bottom, so you know exactly what you're getting.

tail -5000 app.log | python3 lessismore.py -l 2 | llm "why did this crash?"

11x on mixed error logs · 12x on interleaved service logs · 8.9x on ANSI CI logs · 98% on captured pip/tqdm output · ~6 MB/s single-core · 0 dependencies

Why this exists

Three facts about LLM context, and the gap between them is this tool:

1. The expensive part isn't your prompt. Your typed question is ~30 tokens. The log dump you attach is 50,000. Compress the wall, not the question.
2. Machine output is mostly redundancy. ANSI color codes, \r progress redraws, timestamps, venv path spam, the same error line 400 times — noise that looks small on a terminal screen but is real tokens in a capture.
3. BPE tokenizers already compress English (common words are 1 token — you can't out-abbreviate them, we checked). What BPE can't see is repetition across a file or terminal noise. That's the seam this tool mines.

Three payoffs when you pipe through it:

• Context capacity — an hour of log history fits where two minutes did. On a local model, that's the difference between full speed and crawling.
• Prompt-cache longevity — every pass is deterministic: same input, same output, byte for byte. Follow-up questions re-hit the provider cache instead of re-paying for the logs. An ML compressor in the loop would bust the cache on every subtle variation; the regex passes never do.
• Model attention — LLMs lose things in the middle of walls of text. Feed signal, not noise, and the first answer is the right one more often.

Quickstart

pip install .                # from a clone — installs the `lessismore` and `lm` commands
pip install '.[tokens]'      # + tiktoken for exact token stats (else chars/4 estimate)
python3 test_lessismore.py   # should print "ok"

lm dump.log -l 2 > small.log            # file in, file out
docker logs myapp 2>&1 | lm -l 2        # pipe filter
lm big.txt -l 2 --budget 4000           # compress, then hard-cap at ~4k tokens
lm big.txt -l 2 --ml 0.5                # + LLMLingua last mile
lm --serve                              # paste-in demo page on localhost:7777

No install needed either — python3 lessismore.py works the same from a bare clone; it's one stdlib-only file.

from lessismore import compress, count_tokens, budget
small = budget(compress(big_log, level=2), 8000)

Token stats print to stderr, so pipes stay clean.

The dial

Four levels, from byte-cautious to caveman. Pick by content, not by greed.

Level	What it eats	Point it at	Profile
1	whitespace runs, consecutive duplicate lines	code, scripts, anything	structure-safe: indentation and content untouched (whitespace inside string literals still collapses)
2	+ JSON minify, \r redraw collapse, ANSI strip, ISO timestamps, base64/hex blobs, UUIDs, venv paths, scattered-duplicate aliasing, similar-line collapse	logs, CI dumps, traces, tool output	the sweet spot — destroys machine noise, keeps every distinct fact
3	+ filler-phrase stripping ("could you please", "just")	prose, chat history	fine for text, never for strict logic
4	+ two_sticks caveman mode: drops articles/copulas/auxiliaries, never negations or modals	gist-only prose, transcripts	lossy on style, protective of meaning — "do not delete" keeps its not

The crown jewel at level 2 is alias_repeats: ordinary dedupe only sees consecutive repeats, so interleaved multi-service logs sail straight through it. alias_repeats hunts scattered duplicates across the whole file and dictionary-codes them (@1 = ERROR [pool-3] psycopg2... once in a legend, 2-token @1 everywhere else). It's lossless — the legend keeps every line verbatim — and on the interleaved benchmark it's the difference between 54,999 → 34,999 (dedupe alone, 1.6x) and 54,999 → 4,585 (12x).

--budget N is the backstop, not the compressor: after the passes run, it keeps head and tail lines and drops the middle with a [~N tokens omitted] marker. It's the only pass with guaranteed bounded output — use it when the context limit is a hard wall.

The receipts

All reproducible: python3 bench.py (o200k counts via tiktoken).

Sample	Tokens	Why it wins
Mixed error log, 2000 lines	86,914 → 7,895 (11.0x)	timestamp strip unmasks identical lines → dedupe; similar-line collapse catches the numbered stragglers
Interleaved 3-service log, zero consecutive repeats	54,999 → 4,585 (12.0x)	alias_repeats — consecutive dedupe alone managed 1.6x on this input
ANSI-colored CI/docker build log	21,198 → 2,387 (8.9x)	color codes make identical lines look different; strip them and the log collapses
Captured pip/tqdm output	14,483 → 291 (98%)	every overwritten \r progress frame is invisible on screen but real tokens in a capture
pytest failure dump	2,223 → 1,673 (25%)	one site-packages path prefix = 25 tokens → 7
Pretty-printed JSON API response	11,741 → 6,833 (42%)	minify (round-trip verified lossless) + UUID→8-hex squash
Chatty prose, level 4	427 → 264 (38%)	every dropped function word is a whole token; level 3 got 2% on the same text

Throughput: measured ~6 MB/s single-core at level 2 (28 MB log in 4.4s). No model, no network — break-even input size is effectively zero.

What we refused to build (measured so it stays dead)

The graveyard is a feature. Each of these looks clever on paper and loses against a real tokenizer:

• Shorthand codebooks — [fmt:md_tbl+hdr] costs 8 tokens; "Format the output as a markdown table with headers." costs 10. BPE already has English baked in; bracket syntax shreds into off-distribution fragments.
• Word→code recoding ("database" → qx) — common words are already 1 token, random codes cost 2, plus ~4 tokens/entry of codebook tax. You cannot beat a 200k-entry codebook from inside its own encoding. Zipping a zip.
• Known abbreviations (db, fn, env, auth) — measured 1 → 1 tokens. Zero.
• Personalized shorthand skills — scanned 2,917 real typed prompts across 654 transcripts: 2,762 unique, and the repeats were already grunts ("yes", "go"). A model-side decode skill costs ~1k tokens/turn to save ~10. grunts.py keeps the half that works: mine your own history, emit client-side slash-command stubs — expansion before the model sees it is free.
• Digit-masked dedupe (0.0% — progress bars differ in glyphs, not digits), separator shortening (a 78-char ---- is already 1 token), prefix hoisting (one stray line kills it), JSON→TSV / float truncation / pointer squash (real savings, worse trade).

Battle-tested

An adversarial review agent was told to break it and confirmed 14 real failure modes — negation-inverting filler stripping ("was not just the db" → "was not the db"), URLs eaten as base64, crashes on empty and non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated", two_sticks eating "IT" and "US" as function words. Nine fixed with regression tests, four documented below, one wontfix (adversarial in-band marker collision). python3 test_lessismore.py — 30 asserts, no framework.

Where it does nothing (on purpose)

Savings are proportional to redundancy, not size. This removes repetition and machine noise; it cannot compress information, and doesn't pretend to.

Content	Expect	Verdict
Repetitive machine output	5–12x, up to 50x on pathological repeats	the reason this exists
Structured data (JSON, tracebacks)	25–42%	worthwhile, lossless where it fires
Varied prose	~2% (level 3) / 38% lossy (level 4)	gist mode only
Clean code, unique dense text	~0% by design	that's what --ml or truncation is for

--ml RATE bolts on Microsoft's LLMLingua-2 for perplexity pruning as a last mile (pip install llmlingua) — runs after the deterministic passes so you're not paying a classifier model to delete duplicate log lines. Only worth it on multi-KB inputs, and it forfeits the cache-stability guarantee.

Try it in a browser

lm --serve runs a paste-in demo page on http://localhost:7777 — paste your ugliest log, pick a level, watch the token count drop. Stdlib only, binds localhost only.

License

MIT — see LICENSE.

Known tradeoffs

• Levels 2+ assume line order matters but wall-clock timing doesn't. When gaps and deltas are the signal (hang hunting), stay on level 1.
• Output is for LLM consumption, not round-tripping: markdown hard breaks (trailing double-space) and diff context lines don't survive even level 1.
• At levels 3–4, dedupe counts describe the post-stripped text — five differently-phrased "retry the job" lines can legitimately merge.
• In-band markers can collide with input that already contains them; alias_repeats bails out if its own @N markers already appear as lines.