README final pass: current numbers everywhere, rtk calibration, tradeoffs
- hero matches the measured receipts (87k->6.7k mixed log) - rtk's third-party LogDx-CI numbers cited as the lossy end of the curve - hook documented as fail-open; Drain3 comparison matches two-tier design - stale 300x claim corrected (~230x), assert count 48, LogDx bug credited - Known tradeoffs gains templated-readability note, moves above License Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jing
parent
5cad115bb8
commit
6b247ebae2
52
README.md
52
README.md
@ -1,14 +1,14 @@
|
||||
# lessismore 📉
|
||||
|
||||
**Turn 50,000 tokens of log spam into 4,500 tokens of pure signal — before it
|
||||
**Turn 87,000 tokens of log spam into 6,700 tokens of pure signal — before it
|
||||
ever hits your LLM.**
|
||||
|
||||
lessismore is a deterministic compression filter for the bulk text that
|
||||
actually fills context windows: logs, CI output, tracebacks, JSON dumps,
|
||||
captured tool output. Pure stdlib Python. Zero dependencies. Every claim below
|
||||
was measured against the real o200k tokenizer — and every idea that failed the
|
||||
measurement is documented at the bottom, so you know exactly what you're
|
||||
getting.
|
||||
was measured against the real o200k tokenizer, on three public benchmarks as
|
||||
well as synthetic ones — and every idea that failed the measurement is
|
||||
documented in the graveyard section, so you know exactly what you're getting.
|
||||
|
||||
```bash
|
||||
tail -5000 app.log | python3 lessismore.py | llm "why did this crash?"
|
||||
@ -178,7 +178,12 @@ of the 7 misses, most are layout artifacts where templating separates a value
|
||||
from its line). And the answer-quality check: on 12 cases, claude-haiku-4.5
|
||||
diagnosed **12/12 from raw and 12/12 from compressed** — parity at ~2.4x
|
||||
fewer tokens. (Ground-truth caveat inherited from the benchmark: its labels
|
||||
are AI-drafted with single-author verification, n=35.)
|
||||
are AI-drafted with single-author verification, n=35.) For calibration
|
||||
against the field: rtk's most aggressive mode measures 810 tokens/case on
|
||||
this same corpus at a 0.249 diagnosis score
|
||||
([rtk#2012](https://github.com/rtk-ai/rtk/issues/2012)) — a far lossier
|
||||
point on the curve. lessismore's stance is the opposite end: keep the
|
||||
signals, then `--budget` down to whatever your cap is.
|
||||
|
||||
**The equal-budget receipt** — the question every harness actually faces is
|
||||
not "compress or not" but "we cap tool output anyway; is compressing before
|
||||
@ -274,7 +279,8 @@ outputs under 2,000 chars pass through untouched. In
|
||||
|
||||
Because the passes are deterministic, hook-compressed transcripts stay
|
||||
byte-stable across re-runs — the prompt-cache argument above is strongest
|
||||
exactly here.
|
||||
exactly here. The hook fails open: on any input it doesn't recognize it
|
||||
emits nothing, and Claude Code keeps the original output untouched.
|
||||
|
||||
## What we refused to build (measured so it stays dead)
|
||||
|
||||
@ -318,8 +324,9 @@ non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated",
|
||||
regression tests, four documented below, one wontfix (adversarial in-band
|
||||
marker collision). A second review round found the big one: similar-line
|
||||
collapse silently eating distinct status codes — fixed with value summaries
|
||||
and now guarded by the downstream eval. `python3 test_lessismore.py` —
|
||||
42 asserts, no framework.
|
||||
and now guarded by the downstream eval. A third bug (camelCase test names
|
||||
eaten as base64 blobs) was caught by the LogDx-CI benchmark rather than a
|
||||
reviewer. `python3 test_lessismore.py` — 48 asserts, no framework.
|
||||
|
||||
## Where it does nothing (on purpose)
|
||||
|
||||
@ -328,7 +335,7 @@ and machine noise; it cannot compress information, and doesn't pretend to.
|
||||
|
||||
| Content | Expect | Verdict |
|
||||
|---|---|---|
|
||||
| Repetitive machine output | 5–16x, up to 300x on pathological repeats | the reason this exists |
|
||||
| Repetitive machine output | 5–16x, up to ~230x on pathological repeats | the reason this exists |
|
||||
| Real-world CI / system logs | 2–4x typical, up to 10x on template-heavy logs (measured on LogHub, LogChunks, LogDx-CI) | the honest production number |
|
||||
| Structured data (JSON, tracebacks) | 30–64% | worthwhile, lossless where it fires |
|
||||
| Varied prose | ~2% (level 3) / 38% lossy (level 4) | gist mode only |
|
||||
@ -354,9 +361,11 @@ Honest placement, so you can pick the right tool:
|
||||
lessismore doesn't have.
|
||||
- **[Drain3](https://github.com/logpai/Drain3)** — industrial log template
|
||||
mining. `alias_templates` is a deterministic, single-pass, zero-dep
|
||||
Drain-lite (digit-token masking instead of similarity clustering); if you
|
||||
need streaming template state across files or letter-varying parameters
|
||||
(usernames, hostnames), use Drain3.
|
||||
Drain-lite (two-tier hex-run/whole-token masking with exact payoff
|
||||
accounting, instead of similarity clustering); if you need streaming
|
||||
template state across files, or one template spanning letter-varying
|
||||
parameters (we measured that merge and it didn't pay — see the graveyard),
|
||||
use Drain3.
|
||||
- **[LLMLingua-2](https://github.com/microsoft/LLMLingua)** — ML token
|
||||
pruning; wrapped here as the optional `--ml` last mile, *after* the
|
||||
deterministic passes, so you never pay a classifier to delete duplicate
|
||||
@ -376,10 +385,6 @@ the cache-stability argument as a design constraint rather than a patch.
|
||||
ugliest log, pick a level, watch the token count drop. Stdlib only, binds
|
||||
localhost only.
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [LICENSE](LICENSE).
|
||||
|
||||
## Known tradeoffs
|
||||
|
||||
- Levels 2+ assume line *order* matters but wall-clock timing doesn't. When
|
||||
@ -389,10 +394,11 @@ MIT — see [LICENSE](LICENSE).
|
||||
- At levels 3–4, dedupe counts describe the post-stripped text — five
|
||||
differently-phrased "retry the job" lines can legitimately merge.
|
||||
- In-band markers can collide with input that already contains them;
|
||||
`alias_repeats` bails out if its own `@N` markers already appear as lines.
|
||||
`alias_repeats` and `alias_templates` bail out if their own `@N`/`@tN`
|
||||
markers already appear as lines.
|
||||
- Blob squashing eats base64url runs (real JWTs) only when they look like
|
||||
blobs (digits + mixed case); a 64-char kebab-case or snake_case identifier
|
||||
is treated as content and survives.
|
||||
blobs (digits + mixed case, no 16+ letter run); a kebab-case slug or a
|
||||
camelCase test name is treated as content and survives.
|
||||
- Similar-line collapse summarizes varying digits per column
|
||||
(`values 404/429/503`, or a min–max range past 4 distinct); if the digit
|
||||
*runs* per line differ in count, it falls back to a plain count marker.
|
||||
@ -401,3 +407,11 @@ MIT — see [LICENSE](LICENSE).
|
||||
original spacing/alignment is not preserved). Parameters that vary in
|
||||
*letters* (usernames, hostnames) split into separate templates rather than
|
||||
merging — less compression, never confusion.
|
||||
- Reading a templated log requires joining refs back to the legend. Models
|
||||
handle this fine (the evals above are all on templated output, and answers
|
||||
routinely cite `@N` legend entries), but a human skimming the compressed
|
||||
file will prefer the raw one.
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [LICENSE](LICENSE).
|
||||
|
||||
Loading…
Reference in New Issue
Block a user