alias_templates is now two tiers with exact char accounting:
- tier A masks hex/digit runs INSIDE tokens (run-level constants — dates,
IP prefixes, the 2 in ssh2 — inline into the template); groups are
re-costed under whole-token value encoding and take the cheaper
- tier B masks whole digit-bearing tokens on what tier A left, so paths/ids
varying in letters still group (compile/build logs)
- dedupe_similar skeleton is hex-aware but keeps a leading @tN verbatim:
refs from different templates can no longer merge as "similar" — an
earlier build scored 8.9x on LogHub from exactly that silent value
destruction, and v0.3.0's HDFS number had the same phantom; both gone
Measured (bench_real.py): LogHub 3.0x -> 3.9x, LogDx-CI 2.2x -> 2.3x at
unchanged 95.2% critical-signal retention, LogChunks 1.9x -> 2.1x.
Equal-budget receipt added to bench_real logdx: at a 2k cap,
compress-then-truncate retains 46.6% of critical signals vs 24.7% for
plain truncation. Synthetic eval re-run on shipped code: 8/8 at 6.4x.
Graveyard: Drain-style letter-merging measured flat (3.0x -> 3.0x, Linux
regressed) and documented dead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- LogHub samples (real system logs): 3.0x over 1.15M tokens
- LogChunks (797 Travis failures, human-labeled failure chunks): 1.9x over
66M tokens, 78.5% of labeled failure-explaining lines survive verbatim
- LogDx-CI (35 real GitHub Actions failures, arXiv 2605.28876): 2.2x over
15M tokens, 95.2% critical diagnostic signals retained; with --budget
8000, 6,334 tok/case at 77.4% retention; haiku diagnosed 12/12 from
compressed vs 12/12 raw on the 12-case subset
- README: Real logs, real benchmarks section; updated synthetic receipts
(templates lift ANSI CI to 15.9x, synthetic eval total to 6.0x, 8/8
re-verified); honest 2-4x expectation row for production logs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- bench.py now generates every README table row (ANSI CI, pip/tqdm, pytest,
pretty JSON added) and refuses to print chars/4 estimates — reproduced
numbers can no longer silently diverge from the claims
- eval.py: the receipt ratios can't give — 8 seeded failure logs with
planted root causes and red herrings, asked raw vs compressed, graded by
deterministic keyword check. First run (haiku 4.5): raw 6/8, compressed
8/8 at 3.1x fewer tokens; both raw misses were lost-in-the-middle
- README: eval section, related-work section (rtk, Headroom, Drain3,
LLMLingua-2, arXiv 2604.13066), prompt-cache claim sharpened to the
rebuilt-prompt scenario, agent-harness integration (--run, --hook),
re-measured receipts table
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- LICENSE (MIT) and pyproject.toml: pip install . gives lessismore + lm
commands; [tokens] and [ml] extras
- budget(): middle-out hard cap, the only guaranteed-bounded pass; --budget N
runs after compression as the backstop; char-slicing fallback for single
giant lines
- serve(): stdlib paste-in demo page, localhost only; lm --serve
- Scrubbed internal hostnames/IPs from samples and README; benchmarks
re-measured after scrub (interleaved now 54,999 -> 4,585, 12.0x; dedupe
alone 1.6x on same input)
- 30-assert suite, green on system python and tiktoken venv
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
