493c3e912a
- LICENSE (MIT) and pyproject.toml: pip install . gives lessismore + lm commands; [tokens] and [ml] extras - budget(): middle-out hard cap, the only guaranteed-bounded pass; --budget N runs after compression as the backstop; char-slicing fallback for single giant lines - serve(): stdlib paste-in demo page, localhost only; lm --serve - Scrubbed internal hostnames/IPs from samples and README; benchmarks re-measured after scrub (interleaved now 54,999 -> 4,585, 12.0x; dedupe alone 1.6x on same input) - 30-assert suite, green on system python and tiktoken venv Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
201 lines
9.9 KiB
Plaintext
201 lines
9.9 KiB
Plaintext
Metadata-Version: 2.4
|
||
Name: lessismore
|
||
Version: 0.1.0
|
||
Summary: Squeeze text before it hits an LLM — deterministic prompt compression, measured in real tokens
|
||
Author: John King (monsterrobotsoft)
|
||
License: MIT
|
||
Keywords: llm,prompt,compression,tokens,context
|
||
Requires-Python: >=3.8
|
||
Description-Content-Type: text/markdown
|
||
License-File: LICENSE
|
||
Provides-Extra: tokens
|
||
Requires-Dist: tiktoken; extra == "tokens"
|
||
Provides-Extra: ml
|
||
Requires-Dist: llmlingua; extra == "ml"
|
||
Dynamic: license-file
|
||
|
||
# lessismore 📉
|
||
|
||
**Turn 50,000 tokens of log spam into 4,500 tokens of pure signal — before it
|
||
ever hits your LLM.**
|
||
|
||
lessismore is a deterministic compression filter for the bulk text that
|
||
actually fills context windows: logs, CI output, tracebacks, JSON dumps,
|
||
captured tool output. Pure stdlib Python. Zero dependencies. Every claim below
|
||
was measured against the real o200k tokenizer — and every idea that failed the
|
||
measurement is documented at the bottom, so you know exactly what you're
|
||
getting.
|
||
|
||
```bash
|
||
tail -5000 app.log | python3 lessismore.py -l 2 | llm "why did this crash?"
|
||
```
|
||
|
||
**11x** on mixed error logs · **12x** on interleaved service logs ·
|
||
**8.9x** on ANSI CI logs · **98%** on captured pip/tqdm output ·
|
||
**~6 MB/s** single-core · **0** dependencies
|
||
|
||
## Why this exists
|
||
|
||
Three facts about LLM context, and the gap between them is this tool:
|
||
|
||
1. **The expensive part isn't your prompt.** Your typed question is ~30
|
||
tokens. The log dump you attach is 50,000. Compress the wall, not the
|
||
question.
|
||
2. **Machine output is mostly redundancy.** ANSI color codes, `\r` progress
|
||
redraws, timestamps, venv path spam, the same error line 400 times — noise
|
||
that looks small on a terminal screen but is real tokens in a capture.
|
||
3. **BPE tokenizers already compress English** (common words are 1 token —
|
||
you can't out-abbreviate them, we checked). What BPE *can't* see is
|
||
repetition across a file or terminal noise. That's the seam this tool
|
||
mines.
|
||
|
||
Three payoffs when you pipe through it:
|
||
|
||
- **Context capacity** — an hour of log history fits where two minutes did.
|
||
On a local model, that's the difference between full speed and crawling.
|
||
- **Prompt-cache longevity** — every pass is deterministic: same input, same
|
||
output, byte for byte. Follow-up questions re-hit the provider cache
|
||
instead of re-paying for the logs. An ML compressor in the loop would bust
|
||
the cache on every subtle variation; the regex passes never do.
|
||
- **Model attention** — LLMs lose things in the middle of walls of text. Feed
|
||
signal, not noise, and the first answer is the right one more often.
|
||
|
||
## Quickstart
|
||
|
||
```bash
|
||
pip install . # from a clone — installs the `lessismore` and `lm` commands
|
||
pip install '.[tokens]' # + tiktoken for exact token stats (else chars/4 estimate)
|
||
python3 test_lessismore.py # should print "ok"
|
||
```
|
||
|
||
```bash
|
||
lm dump.log -l 2 > small.log # file in, file out
|
||
docker logs myapp 2>&1 | lm -l 2 # pipe filter
|
||
lm big.txt -l 2 --budget 4000 # compress, then hard-cap at ~4k tokens
|
||
lm big.txt -l 2 --ml 0.5 # + LLMLingua last mile
|
||
lm --serve # paste-in demo page on localhost:7777
|
||
```
|
||
|
||
No install needed either — `python3 lessismore.py` works the same from a bare
|
||
clone; it's one stdlib-only file.
|
||
|
||
```python
|
||
from lessismore import compress, count_tokens, budget
|
||
small = budget(compress(big_log, level=2), 8000)
|
||
```
|
||
|
||
Token stats print to stderr, so pipes stay clean.
|
||
|
||
## The dial
|
||
|
||
Four levels, from byte-cautious to caveman. Pick by content, not by greed.
|
||
|
||
| Level | What it eats | Point it at | Profile |
|
||
|---|---|---|---|
|
||
| **1** | whitespace runs, consecutive duplicate lines | code, scripts, anything | structure-safe: indentation and content untouched (whitespace *inside string literals* still collapses) |
|
||
| **2** | + JSON minify, `\r` redraw collapse, ANSI strip, ISO timestamps, base64/hex blobs, UUIDs, venv paths, scattered-duplicate aliasing, similar-line collapse | logs, CI dumps, traces, tool output | **the sweet spot** — destroys machine noise, keeps every distinct fact |
|
||
| **3** | + filler-phrase stripping ("could you please", "just") | prose, chat history | fine for text, never for strict logic |
|
||
| **4** | + `two_sticks` caveman mode: drops articles/copulas/auxiliaries, never negations or modals | gist-only prose, transcripts | lossy on style, protective of meaning — "do **not** delete" keeps its *not* |
|
||
|
||
The crown jewel at level 2 is `alias_repeats`: ordinary dedupe only sees
|
||
*consecutive* repeats, so interleaved multi-service logs sail straight through
|
||
it. `alias_repeats` hunts scattered duplicates across the whole file and
|
||
dictionary-codes them (`@1 = ERROR [pool-3] psycopg2...` once in a legend,
|
||
2-token `@1` everywhere else). It's lossless — the legend keeps every line
|
||
verbatim — and on the interleaved benchmark it's the difference between
|
||
54,999 → 34,999 (dedupe alone, 1.6x) and 54,999 → 4,585 (**12x**).
|
||
|
||
`--budget N` is the backstop, not the compressor: after the passes run, it
|
||
keeps head and tail lines and drops the middle with a `[~N tokens omitted]`
|
||
marker. It's the only pass with *guaranteed* bounded output — use it when the
|
||
context limit is a hard wall.
|
||
|
||
## The receipts
|
||
|
||
All reproducible: `python3 bench.py` (o200k counts via tiktoken).
|
||
|
||
| Sample | Tokens | Why it wins |
|
||
|---|---|---|
|
||
| Mixed error log, 2000 lines | 86,914 → 7,895 (**11.0x**) | timestamp strip unmasks identical lines → dedupe; similar-line collapse catches the numbered stragglers |
|
||
| Interleaved 3-service log, zero consecutive repeats | 54,999 → 4,585 (**12.0x**) | `alias_repeats` — consecutive dedupe alone managed 1.6x on this input |
|
||
| ANSI-colored CI/docker build log | 21,198 → 2,387 (**8.9x**) | color codes make identical lines look different; strip them and the log collapses |
|
||
| Captured pip/tqdm output | 14,483 → 291 (**98%**) | every overwritten `\r` progress frame is invisible on screen but real tokens in a capture |
|
||
| pytest failure dump | 2,223 → 1,673 (25%) | one site-packages path prefix = 25 tokens → 7 |
|
||
| Pretty-printed JSON API response | 11,741 → 6,833 (42%) | minify (round-trip verified lossless) + UUID→8-hex squash |
|
||
| Chatty prose, level 4 | 427 → 264 (38%) | every dropped function word is a whole token; level 3 got 2% on the same text |
|
||
|
||
Throughput: measured **~6 MB/s single-core** at level 2 (28 MB log in 4.4s).
|
||
No model, no network — break-even input size is effectively zero.
|
||
|
||
## What we refused to build (measured so it stays dead)
|
||
|
||
The graveyard is a feature. Each of these looks clever on paper and loses
|
||
against a real tokenizer:
|
||
|
||
- **Shorthand codebooks** — `[fmt:md_tbl+hdr]` costs **8 tokens**; "Format
|
||
the output as a markdown table with headers." costs **10**. BPE already has
|
||
English baked in; bracket syntax shreds into off-distribution fragments.
|
||
- **Word→code recoding** ("database" → `qx`) — common words are already 1
|
||
token, random codes cost 2, plus ~4 tokens/entry of codebook tax. You
|
||
cannot beat a 200k-entry codebook from inside its own encoding. Zipping a
|
||
zip.
|
||
- **Known abbreviations** (db, fn, env, auth) — measured 1 → 1 tokens. Zero.
|
||
- **Personalized shorthand skills** — scanned 2,917 real typed prompts across
|
||
654 transcripts: 2,762 unique, and the repeats were already grunts ("yes",
|
||
"go"). A model-side decode skill costs ~1k tokens/turn to save ~10.
|
||
`grunts.py` keeps the half that works: mine your own history, emit
|
||
*client-side* slash-command stubs — expansion before the model sees it is
|
||
free.
|
||
- **Digit-masked dedupe** (0.0% — progress bars differ in glyphs, not
|
||
digits), **separator shortening** (a 78-char `----` is already 1 token),
|
||
**prefix hoisting** (one stray line kills it), **JSON→TSV / float
|
||
truncation / pointer squash** (real savings, worse trade).
|
||
|
||
## Battle-tested
|
||
|
||
An adversarial review agent was told to break it and confirmed 14 real
|
||
failure modes — negation-inverting filler stripping ("was **not just** the
|
||
db" → "was **not** the db"), URLs eaten as base64, crashes on empty and
|
||
non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated",
|
||
`two_sticks` eating "IT" and "US" as function words. Nine fixed with
|
||
regression tests, four documented below, one wontfix (adversarial in-band
|
||
marker collision). `python3 test_lessismore.py` — 30 asserts, no framework.
|
||
|
||
## Where it does nothing (on purpose)
|
||
|
||
Savings are proportional to **redundancy, not size**. This removes repetition
|
||
and machine noise; it cannot compress information, and doesn't pretend to.
|
||
|
||
| Content | Expect | Verdict |
|
||
|---|---|---|
|
||
| Repetitive machine output | 5–12x, up to 50x on pathological repeats | the reason this exists |
|
||
| Structured data (JSON, tracebacks) | 25–42% | worthwhile, lossless where it fires |
|
||
| Varied prose | ~2% (level 3) / 38% lossy (level 4) | gist mode only |
|
||
| Clean code, unique dense text | ~0% by design | that's what `--ml` or truncation is for |
|
||
|
||
`--ml RATE` bolts on Microsoft's LLMLingua-2 for perplexity pruning as a
|
||
last mile (`pip install llmlingua`) — runs *after* the deterministic passes so
|
||
you're not paying a classifier model to delete duplicate log lines. Only
|
||
worth it on multi-KB inputs, and it forfeits the cache-stability guarantee.
|
||
|
||
## Try it in a browser
|
||
|
||
`lm --serve` runs a paste-in demo page on `http://localhost:7777` — paste your
|
||
ugliest log, pick a level, watch the token count drop. Stdlib only, binds
|
||
localhost only.
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE).
|
||
|
||
## Known tradeoffs
|
||
|
||
- Levels 2+ assume line *order* matters but wall-clock timing doesn't. When
|
||
gaps and deltas are the signal (hang hunting), stay on level 1.
|
||
- Output is for LLM consumption, not round-tripping: markdown hard breaks
|
||
(trailing double-space) and diff context lines don't survive even level 1.
|
||
- At levels 3–4, dedupe counts describe the post-stripped text — five
|
||
differently-phrased "retry the job" lines can legitimately merge.
|
||
- In-band markers can collide with input that already contains them;
|
||
`alias_repeats` bails out if its own `@N` markers already appear as lines.
|