Benchmarking on real corpora (LogHub) exposed that synthetic logs flatter
the compressor: real logs repeat templates, not exact lines, and level 2
managed only 1.5x. Changes, each measured:
- alias_templates: lines differing only in digit-bearing tokens become one
legend template (@t1 = Failed password for root from <*> port <*> ssh2)
plus per-line values; constant digit tokens (ssh2) inline into the
template. Deterministic, single-pass, in-band bail-out, same payoff bar
as alias_repeats. LogHub corpus: 1.8x -> 3.0x.
- four field-observed timestamp formats (HDFS yymmdd hhmmss, BGL RAS
stamps, epoch+date pairs, HealthApp) — LogHub: 1.5x -> 1.8x
- blob squash: long alpha runs (16+ letters) exempt the -/_ branch — a
camelCase test name was being eaten as base64; caught by LogDx-CI,
regression-tested
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- bench.py now generates every README table row (ANSI CI, pip/tqdm, pytest,
pretty JSON added) and refuses to print chars/4 estimates — reproduced
numbers can no longer silently diverge from the claims
- eval.py: the receipt ratios can't give — 8 seeded failure logs with
planted root causes and red herrings, asked raw vs compressed, graded by
deterministic keyword check. First run (haiku 4.5): raw 6/8, compressed
8/8 at 3.1x fewer tokens; both raw misses were lost-in-the-middle
- README: eval section, related-work section (rtk, Headroom, Drain3,
LLMLingua-2, arXiv 2604.13066), prompt-cache claim sharpened to the
rebuilt-prompt scenario, agent-harness integration (--run, --hook),
re-measured receipts table
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- dedupe_similar: summarize varying digits in the marker (values 404/429/503
or min-max) — distinct status codes/ports no longer vanish silently
- minify_json: also fires on pretty JSON embedded in other text (the
captured-tool-output shape), via raw_decode at line-breaking braces
- squash_blobs: admit base64url (- and _) so real JWTs squash, guarded by
digits+mixed-case so kebab/snake identifiers survive
- strip_timestamps: syslog and nginx CLF formats join ISO-8601
- level 2 is now the default (the pitch, the examples, and the sweet spot
all said 2; level 1 stays the explicit code-safe opt-down)
- lm --run CMD: compress a command's output, keep its exit code
- lm --hook: zero-dep Claude Code PostToolUse hook (reads hook JSON on
stdin, emits updatedToolOutput; <2k-char outputs pass through)
- tests: 42 asserts incl. level-2 idempotency invariant, value-summary
regression, embedded JSON, JWT/slug boundary, --run exit passthrough
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- LICENSE (MIT) and pyproject.toml: pip install . gives lessismore + lm
commands; [tokens] and [ml] extras
- budget(): middle-out hard cap, the only guaranteed-bounded pass; --budget N
runs after compression as the backstop; char-slicing fallback for single
giant lines
- serve(): stdlib paste-in demo page, localhost only; lm --serve
- Scrubbed internal hostnames/IPs from samples and README; benchmarks
re-measured after scrub (interleaved now 54,999 -> 4,585, 12.0x; dedupe
alone 1.6x on same input)
- 30-assert suite, green on system python and tiktoken venv
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
