diff --git a/README.md b/README.md index 2f331f9..658ab6b 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,14 @@ # lessismore πŸ“‰ -**Turn 50,000 tokens of log spam into 4,500 tokens of pure signal β€” before it +**Turn 87,000 tokens of log spam into 6,700 tokens of pure signal β€” before it ever hits your LLM.** lessismore is a deterministic compression filter for the bulk text that actually fills context windows: logs, CI output, tracebacks, JSON dumps, captured tool output. Pure stdlib Python. Zero dependencies. Every claim below -was measured against the real o200k tokenizer β€” and every idea that failed the -measurement is documented at the bottom, so you know exactly what you're -getting. +was measured against the real o200k tokenizer, on three public benchmarks as +well as synthetic ones β€” and every idea that failed the measurement is +documented in the graveyard section, so you know exactly what you're getting. ```bash tail -5000 app.log | python3 lessismore.py | llm "why did this crash?" @@ -178,7 +178,12 @@ of the 7 misses, most are layout artifacts where templating separates a value from its line). And the answer-quality check: on 12 cases, claude-haiku-4.5 diagnosed **12/12 from raw and 12/12 from compressed** β€” parity at ~2.4x fewer tokens. (Ground-truth caveat inherited from the benchmark: its labels -are AI-drafted with single-author verification, n=35.) +are AI-drafted with single-author verification, n=35.) For calibration +against the field: rtk's most aggressive mode measures 810 tokens/case on +this same corpus at a 0.249 diagnosis score +([rtk#2012](https://github.com/rtk-ai/rtk/issues/2012)) β€” a far lossier +point on the curve. lessismore's stance is the opposite end: keep the +signals, then `--budget` down to whatever your cap is. **The equal-budget receipt** β€” the question every harness actually faces is not "compress or not" but "we cap tool output anyway; is compressing before @@ -274,7 +279,8 @@ outputs under 2,000 chars pass through untouched. In Because the passes are deterministic, hook-compressed transcripts stay byte-stable across re-runs β€” the prompt-cache argument above is strongest -exactly here. +exactly here. The hook fails open: on any input it doesn't recognize it +emits nothing, and Claude Code keeps the original output untouched. ## What we refused to build (measured so it stays dead) @@ -318,8 +324,9 @@ non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated", regression tests, four documented below, one wontfix (adversarial in-band marker collision). A second review round found the big one: similar-line collapse silently eating distinct status codes β€” fixed with value summaries -and now guarded by the downstream eval. `python3 test_lessismore.py` β€” -42 asserts, no framework. +and now guarded by the downstream eval. A third bug (camelCase test names +eaten as base64 blobs) was caught by the LogDx-CI benchmark rather than a +reviewer. `python3 test_lessismore.py` β€” 48 asserts, no framework. ## Where it does nothing (on purpose) @@ -328,7 +335,7 @@ and machine noise; it cannot compress information, and doesn't pretend to. | Content | Expect | Verdict | |---|---|---| -| Repetitive machine output | 5–16x, up to 300x on pathological repeats | the reason this exists | +| Repetitive machine output | 5–16x, up to ~230x on pathological repeats | the reason this exists | | Real-world CI / system logs | 2–4x typical, up to 10x on template-heavy logs (measured on LogHub, LogChunks, LogDx-CI) | the honest production number | | Structured data (JSON, tracebacks) | 30–64% | worthwhile, lossless where it fires | | Varied prose | ~2% (level 3) / 38% lossy (level 4) | gist mode only | @@ -354,9 +361,11 @@ Honest placement, so you can pick the right tool: lessismore doesn't have. - **[Drain3](https://github.com/logpai/Drain3)** β€” industrial log template mining. `alias_templates` is a deterministic, single-pass, zero-dep - Drain-lite (digit-token masking instead of similarity clustering); if you - need streaming template state across files or letter-varying parameters - (usernames, hostnames), use Drain3. + Drain-lite (two-tier hex-run/whole-token masking with exact payoff + accounting, instead of similarity clustering); if you need streaming + template state across files, or one template spanning letter-varying + parameters (we measured that merge and it didn't pay β€” see the graveyard), + use Drain3. - **[LLMLingua-2](https://github.com/microsoft/LLMLingua)** β€” ML token pruning; wrapped here as the optional `--ml` last mile, *after* the deterministic passes, so you never pay a classifier to delete duplicate @@ -376,10 +385,6 @@ the cache-stability argument as a design constraint rather than a patch. ugliest log, pick a level, watch the token count drop. Stdlib only, binds localhost only. -## License - -MIT β€” see [LICENSE](LICENSE). - ## Known tradeoffs - Levels 2+ assume line *order* matters but wall-clock timing doesn't. When @@ -389,10 +394,11 @@ MIT β€” see [LICENSE](LICENSE). - At levels 3–4, dedupe counts describe the post-stripped text β€” five differently-phrased "retry the job" lines can legitimately merge. - In-band markers can collide with input that already contains them; - `alias_repeats` bails out if its own `@N` markers already appear as lines. + `alias_repeats` and `alias_templates` bail out if their own `@N`/`@tN` + markers already appear as lines. - Blob squashing eats base64url runs (real JWTs) only when they look like - blobs (digits + mixed case); a 64-char kebab-case or snake_case identifier - is treated as content and survives. + blobs (digits + mixed case, no 16+ letter run); a kebab-case slug or a + camelCase test name is treated as content and survives. - Similar-line collapse summarizes varying digits per column (`values 404/429/503`, or a min–max range past 4 distinct); if the digit *runs* per line differ in count, it falls back to a plain count marker. @@ -401,3 +407,11 @@ MIT β€” see [LICENSE](LICENSE). original spacing/alignment is not preserved). Parameters that vary in *letters* (usernames, hostnames) split into separate templates rather than merging β€” less compression, never confusion. +- Reading a templated log requires joining refs back to the legend. Models + handle this fine (the evals above are all on templated output, and answers + routinely cite `@N` legend entries), but a human skimming the compressed + file will prefer the raw one. + +## License + +MIT β€” see [LICENSE](LICENSE).