monster/lessismore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Receipts upgrade: full bench coverage, downstream root-cause eval, 0.2.0

Browse Source 
- bench.py now generates every README table row (ANSI CI, pip/tqdm, pytest,
  pretty JSON added) and refuses to print chars/4 estimates — reproduced
  numbers can no longer silently diverge from the claims
- eval.py: the receipt ratios can't give — 8 seeded failure logs with
  planted root causes and red herrings, asked raw vs compressed, graded by
  deterministic keyword check. First run (haiku 4.5): raw 6/8, compressed
  8/8 at 3.1x fewer tokens; both raw misses were lost-in-the-middle
- README: eval section, related-work section (rtk, Headroom, Drain3,
  LLMLingua-2, arXiv 2604.13066), prompt-cache claim sharpened to the
  rebuilt-prompt scenario, agent-harness integration (--run, --hook),
  re-measured receipts table

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jing 2026-07-07 16:36:56 +10:00
parent cd9abf26ce
commit 7962887057
4 changed files with 451 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

173
README.md
View File

@ -11,12 +11,16 @@ measurement is documented at the bottom, so you know exactly what you're
getting. getting.
```bash ```bash
tail -5000 app.log | python3 lessismore.py -l 2 | llm "why did this crash?" tail -5000 app.log | python3 lessismore.py | llm "why did this crash?"
``` ```
**11x** on mixed error logs · **12x** on interleaved service logs · **11x** on mixed error logs · **12x** on interleaved service logs ·
**8.9x** on ANSI CI logs · **98%** on captured pip/tqdm output · **11.8x** on ANSI CI logs · **97%** on captured pip/tqdm output ·
**~6 MB/s** single-core · **0** dependencies **~6–12 MB/s** single-core · **0** dependencies
And the part a compression ratio can't prove: in the root-cause eval
([eval.py](eval.py)), the model found the planted root cause in **8/8**
compressed logs vs **6/8** raw — smaller *and* more often right.
## Why this exists ## Why this exists
@ -37,10 +41,15 @@ Three payoffs when you pipe through it:
- **Context capacity** — an hour of log history fits where two minutes did. - **Context capacity** — an hour of log history fits where two minutes did.
On a local model, that's the difference between full speed and crawling. On a local model, that's the difference between full speed and crawling.
- **Prompt-cache longevity** — every pass is deterministic: same input, same - **Prompt-cache longevity** — every pass is deterministic and idempotent:
output, byte for byte. Follow-up questions re-hit the provider cache same input, same output, byte for byte (it's a test invariant). This
instead of re-paying for the logs. An ML compressor in the loop would bust matters wherever a prompt is *re-generated* from source each call — a CI
the cache on every subtle variation; the regex passes never do. assistant re-reading the same log, RAG context rebuilt per request, a hook
re-running on every tool call. A deterministic pass produces the identical
prefix every time, so the provider cache hits; an ML compressor in the
loop busts it on every subtle variation. (Within a single chat, follow-ups
hit the cache regardless — the transcript is append-only. The claim is
about rebuilt prompts, not chat turns.)
- **Model attention** — LLMs lose things in the middle of walls of text. Feed - **Model attention** — LLMs lose things in the middle of walls of text. Feed
signal, not noise, and the first answer is the right one more often. signal, not noise, and the first answer is the right one more often.
@ -53,10 +62,12 @@ python3 test_lessismore.py # should print "ok"
``` ```
```bash ```bash
lm dump.log -l 2 > small.log # file in, file out lm dump.log > small.log # file in, file out (level 2 is the default)
docker logs myapp 2>&1 | lm -l 2 # pipe filter docker logs myapp 2>&1 | lm # pipe filter
lm big.txt -l 2 --budget 4000 # compress, then hard-cap at ~4k tokens lm --run "pytest -x" # run a noisy command, print it compressed,
lm big.txt -l 2 --ml 0.5 # + LLMLingua last mile # keep its exit code (pipes can't)
lm big.txt --budget 4000 # compress, then hard-cap at ~4k tokens
lm big.txt --ml 0.5 # + LLMLingua last mile
lm --serve # paste-in demo page on localhost:7777 lm --serve # paste-in demo page on localhost:7777
``` ```
@ -77,10 +88,16 @@ Four levels, from byte-cautious to caveman. Pick by content, not by greed.
| Level | What it eats | Point it at | Profile | | Level | What it eats | Point it at | Profile |
|---|---|---|---| |---|---|---|---|
| **1** | whitespace runs, consecutive duplicate lines | code, scripts, anything | structure-safe: indentation and content untouched (whitespace *inside string literals* still collapses) | | **1** | whitespace runs, consecutive duplicate lines | code, scripts, anything | structure-safe: indentation and content untouched (whitespace *inside string literals* still collapses) |
| **2** | + JSON minify, `\r` redraw collapse, ANSI strip, ISO timestamps, base64/hex blobs, UUIDs, venv paths, scattered-duplicate aliasing, similar-line collapse | logs, CI dumps, traces, tool output | **the sweet spot** — destroys machine noise, keeps every distinct fact | | **2** | + JSON minify (whole-input *and* embedded pretty blocks), `\r` redraw collapse, ANSI strip, ISO/syslog/nginx timestamps, base64/base64url/hex blobs, UUIDs, venv paths, scattered-duplicate aliasing, similar-line collapse with value summaries | logs, CI dumps, traces, tool output | **the sweet spot and the default** — destroys machine noise, keeps every distinct fact |
| **3** | + filler-phrase stripping ("could you please", "just") | prose, chat history | fine for text, never for strict logic | | **3** | + filler-phrase stripping ("could you please", "just") | prose, chat history | fine for text, never for strict logic |
| **4** | + `two_sticks` caveman mode: drops articles/copulas/auxiliaries, never negations or modals | gist-only prose, transcripts | lossy on style, protective of meaning — "do **not** delete" keeps its *not* | | **4** | + `two_sticks` caveman mode: drops articles/copulas/auxiliaries, never negations or modals | gist-only prose, transcripts | lossy on style, protective of meaning — "do **not** delete" keeps its *not* |
"Keeps every distinct fact" is enforced, not hoped: when similar lines
collapse, the digits that varied are summarized in the marker —
`[3 similar lines omitted; values 404/429/503]` — because sometimes the
digits (status codes, ports, exit codes) *are* the diagnosis. An earlier
version silently ate them; the adversarial eval below is what caught it.
The crown jewel at level 2 is `alias_repeats`: ordinary dedupe only sees The crown jewel at level 2 is `alias_repeats`: ordinary dedupe only sees
*consecutive* repeats, so interleaved multi-service logs sail straight through *consecutive* repeats, so interleaved multi-service logs sail straight through
it. `alias_repeats` hunts scattered duplicates across the whole file and it. `alias_repeats` hunts scattered duplicates across the whole file and
@ -96,20 +113,95 @@ context limit is a hard wall.
## The receipts ## The receipts
All reproducible: `python3 bench.py` (o200k counts via tiktoken). Every row reproducible from one command: `python3 bench.py` (o200k counts via
tiktoken — it refuses to print estimate numbers, so what you measure is what
this table claims).
| Sample | Tokens | Why it wins | | Sample | Tokens | Why it wins |
|---|---|---| |---|---|---|
| Mixed error log, 2000 lines | 86,914 → 7,895 (**11.0x**) | timestamp strip unmasks identical lines → dedupe; similar-line collapse catches the numbered stragglers | | Mixed error log, 2000 lines | 86,914 → 7,895 (**11.0x**) | timestamp strip unmasks identical lines → dedupe; similar-line collapse catches the numbered stragglers |
| Interleaved 3-service log, zero consecutive repeats | 54,999 → 4,585 (**12.0x**) | `alias_repeats` — consecutive dedupe alone managed 1.6x on this input | | Interleaved 3-service log, zero consecutive repeats | 54,999 → 4,585 (**12.0x**) | `alias_repeats` — consecutive dedupe alone managed 1.6x on this input |
| ANSI-colored CI/docker build log | 21,198 → 2,387 (**8.9x**) | color codes make identical lines look different; strip them and the log collapses | | ANSI-colored CI log | 10,440 → 888 (**11.8x**) | color codes make identical lines look different; strip them and the log collapses |
| Captured pip/tqdm output | 14,483 → 291 (**98%**) | every overwritten `\r` progress frame is invisible on screen but real tokens in a capture | | Captured pip/tqdm output | 10,465 → 265 (**97%**) | every overwritten `\r` progress frame is invisible on screen but real tokens in a capture |
| pytest failure dump | 2,223 → 1,673 (25%) | one site-packages path prefix = 25 tokens → 7 | | pytest failure dump | 732 → 515 (30%) | one site-packages path prefix = 25 tokens → 7 |
| Pretty-printed JSON API response | 11,741 → 6,833 (42%) | minify (round-trip verified lossless) + UUID→8-hex squash | | Pretty-printed JSON API response | 8,932 → 3,234 (**64%**) | minify (round-trip verified lossless) + UUID→8-hex squash |
| Chatty prose, level 4 | 427 → 264 (38%) | every dropped function word is a whole token; level 3 got 2% on the same text | | Chatty prose, level 4 | 427 → 264 (38%) | every dropped function word is a whole token; level 3 got 2% on the same text |
Throughput: measured **~6 MB/s single-core** at level 2 (28 MB log in 4.4s). Throughput: measured **6–12 MB/s single-core** at level 2 (20 MB log in
No model, no network — break-even input size is effectively zero. 1.7s). No model, no network — break-even input size is effectively zero.
## Does the model still get the answer?
A compression ratio proves the log got smaller, not that the signal
survived. So this is the eval that matters, and it's in the repo:
[eval.py](eval.py) generates eight seeded failure logs, each with one planted
root cause buried in machine noise **plus a red herring** (e.g. a cascade of
connection errors *caused by* an OOM kill 400 lines earlier), asks a model
"what's the root cause?" on the raw and the compressed version, and grades
the answer with a deterministic keyword check.
Result (claude-haiku-4.5, 2026-07-07, one run; this run went through the
Claude Code agent harness with the same single-turn prompts — `eval.py`'s
default runner is a logged-in `claude -p`. Rerun it yourself):
| Scenario | Raw tokens | Compressed | Raw | Compressed |
|---|---|---|---|---|
| OOM kill buried mid-log, connection-error red herring | 38,777 | 157 (**247x**) | ✗ blamed the red herring | ✓ found the OOM |
| 429 rate-limit burst hidden in a wall of status codes | 46,899 | 6,056 | ✗ "service overwhelmed" | ✓ named the 429s |
| Disk-full scattered through 3 interleaved services | 38,616 | 12,487 | ✓ | ✓ |
| One failed assertion in 1,400 ANSI test lines | 46,812 | 21,563 | ✓ | ✓ |
| Dependency conflict under progress-bar walls | 16,304 | 446 | ✓ | ✓ |
| Expired TLS cert in an nginx access-log wall | 56,696 | 34,474 | ✓ | ✓ |
| Missing env var in a crash-loop traceback | 7,680 | 1,610 | ✓ | ✓ |
| Java deadlock in a 40-thread dump | 8,157 | 8,041 (1.0x) | ✓ | ✓ |
| **Total** | **259,941** | **84,834 (3.1x)** | **6/8** | **8/8** |
The two raw failures are the lost-in-the-middle effect this tool exists to
counter: the model latched onto the loud symptom cascade and never surfaced
the quiet cause. Compressed, the cause is impossible to miss. The honest
caveats: n=1 run per condition, synthetic logs authored by this project, one
model — which is exactly why the harness ships in the repo with fixed seeds.
`python3 eval.py --dry` costs nothing and shows the scenarios; point
`LM_EVAL_CMD` at any prompt-on-stdin CLI to grade your own model.
## Where the tokens actually are: agent harnesses
A human remembering to pipe is the demo. The 50k-token walls of 2026 enter
context windows as **tool output inside agent harnesses**, so lessismore
ships two integrations:
**`lm --run`** — run the noisy command through the compressor and keep its
exit code (a shell pipe can't do that without pipefail games):
```bash
lm --run "pytest -x" # agent sees 500 tokens, not 20,000
lm --run "docker build ."
```
Tell your agent about it once (CLAUDE.md: *"run noisy commands through
`lm --run`"*) and every test run gets cheaper.
**`lm --hook`** — a Claude Code `PostToolUse` hook that compresses Bash
output before it enters the model's context. Zero dependencies, no `jq`;
outputs under 2,000 chars pass through untouched. In
`.claude/settings.json`:
```json
{
"hooks": {
"PostToolUse": [
{
"matcher": "Bash",
"hooks": [{ "type": "command", "command": "lm --hook" }]
}
]
}
}
```
Because the passes are deterministic, hook-compressed transcripts stay
byte-stable across re-runs — the prompt-cache argument above is strongest
exactly here.
## What we refused to build (measured so it stays dead) ## What we refused to build (measured so it stays dead)
@ -143,7 +235,10 @@ db" → "was **not** the db"), URLs eaten as base64, crashes on empty and
non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated", non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated",
`two_sticks` eating "IT" and "US" as function words. Nine fixed with `two_sticks` eating "IT" and "US" as function words. Nine fixed with
regression tests, four documented below, one wontfix (adversarial in-band regression tests, four documented below, one wontfix (adversarial in-band
marker collision). `python3 test_lessismore.py` — 30 asserts, no framework. marker collision). A second review round found the big one: similar-line
collapse silently eating distinct status codes — fixed with value summaries
and now guarded by the downstream eval. `python3 test_lessismore.py` —
42 asserts, no framework.
## Where it does nothing (on purpose) ## Where it does nothing (on purpose)
@ -153,7 +248,7 @@ and machine noise; it cannot compress information, and doesn't pretend to.
| Content | Expect | Verdict | | Content | Expect | Verdict |
|---|---|---| |---|---|---|
| Repetitive machine output | 5–12x, up to 50x on pathological repeats | the reason this exists | | Repetitive machine output | 5–12x, up to 50x on pathological repeats | the reason this exists |
| Structured data (JSON, tracebacks) | 25–42% | worthwhile, lossless where it fires | | Structured data (JSON, tracebacks) | 30–64% | worthwhile, lossless where it fires |
| Varied prose | ~2% (level 3) / 38% lossy (level 4) | gist mode only | | Varied prose | ~2% (level 3) / 38% lossy (level 4) | gist mode only |
| Clean code, unique dense text | ~0% by design | that's what `--ml` or truncation is for | | Clean code, unique dense text | ~0% by design | that's what `--ml` or truncation is for |
@ -162,6 +257,36 @@ last mile (`pip install llmlingua`) — runs *after* the deterministic passes so
you're not paying a classifier model to delete duplicate log lines. Only you're not paying a classifier model to delete duplicate log lines. Only
worth it on multi-KB inputs, and it forfeits the cache-stability guarantee. worth it on multi-KB inputs, and it forfeits the cache-stability guarantee.
## Related work (know the river you're panning)
Honest placement, so you can pick the right tool:
- **[rtk](https://github.com/rtk-ai/rtk)** — Rust proxy with per-command
filters (ANSI strip, dedupe, failures-only test output), hooks into agent
CLIs. Command-aware and aggressive where lessismore is generic text passes
and lossless-leaning: rtk decides what you need to see; lessismore removes
only what is provably redundant and keeps a legend.
- **[Headroom](https://github.com/chopratejas/headroom)** — the heavyweight
tool-output compressor (Python, proxy + MCP). Has ML models in the loop,
so it needs a cache-alignment component to patch the non-determinism
lessismore doesn't have.
- **[Drain3](https://github.com/logpai/Drain3)** — industrial log template
mining. `alias_repeats` + `dedupe_similar` are a single-pass, zero-dep
approximation of it; if you need streaming template state across files,
use Drain3.
- **[LLMLingua-2](https://github.com/microsoft/LLMLingua)** — ML token
pruning; wrapped here as the optional `--ml` last mile, *after* the
deterministic passes, so you never pay a classifier to delete duplicate
log lines.
- **[Dictionary-encoding prompt compression](https://arxiv.org/abs/2604.13066)**
(2026) — independently validates the `@1 = line` legend technique
academically (≥0.99 fidelity on log benchmarks); no code released.
lessismore is, in effect, a reference implementation.
What none of them package together — and the reason this exists — is the
combination: pure-stdlib, deterministic end to end, tokenizer-measured, with
the cache-stability argument as a design constraint rather than a patch.
## Try it in a browser ## Try it in a browser
`lm --serve` runs a paste-in demo page on `http://localhost:7777` — paste your `lm --serve` runs a paste-in demo page on `http://localhost:7777` — paste your
@ -182,3 +307,9 @@ MIT — see [LICENSE](LICENSE).
differently-phrased "retry the job" lines can legitimately merge. differently-phrased "retry the job" lines can legitimately merge.
- In-band markers can collide with input that already contains them; - In-band markers can collide with input that already contains them;
`alias_repeats` bails out if its own `@N` markers already appear as lines. `alias_repeats` bails out if its own `@N` markers already appear as lines.
- Blob squashing eats base64url runs (real JWTs) only when they look like
blobs (digits + mixed case); a 64-char kebab-case or snake_case identifier
is treated as content and survives.
- Similar-line collapse summarizes varying digits per column
(`values 404/429/503`, or a min–max range past 4 distinct); if the digit
*runs* per line differ in count, it falls back to a plain count marker.

66
bench.py
View File

@ -1,7 +1,19 @@
"""Reproduce the README benchmark numbers. pip install tiktoken for exact counts.""" """Reproduce the README benchmark numbers — every row of the receipts table.
import random
from lessismore import compress, count_tokens Requires tiktoken (pip install tiktoken): the README numbers are o200k counts,
and printing chars/4 estimates here would let the receipts silently drift from
the claims. Pass --estimate if you really want the guesses.
"""
import json
import random
import sys
from lessismore import compress, count_tokens, _encoder
if _encoder() is None and "--estimate" not in sys.argv:
sys.exit("bench.py: tiktoken not installed — these would be chars/4 GUESSES,\n"
"not the o200k-measured numbers the README claims.\n"
" pip install tiktoken (or pass --estimate to see the guesses anyway)")
def bench(name, text, level): def bench(name, text, level):
@ -23,7 +35,6 @@ lines.insert(500, "2026-07-07T10:08:20.123Z DEBUG auth token=" + "eyJhbGciOiJIUz
bench("mixed error log", "\n".join(lines), 2) bench("mixed error log", "\n".join(lines), 2)
# interleaved log: three services alternating — zero consecutive runs, dedupe-proof # interleaved log: three services alternating — zero consecutive runs, dedupe-proof
random.seed(2)
msgs = ["ERROR [pool-3] psycopg2.OperationalError: connection to server at " msgs = ["ERROR [pool-3] psycopg2.OperationalError: connection to server at "
"10.0.0.5 failed: Connection refused", "10.0.0.5 failed: Connection refused",
"WARN [redis-1] retry queue depth exceeded threshold, backing off 500ms", "WARN [redis-1] retry queue depth exceeded threshold, backing off 500ms",
@ -31,6 +42,53 @@ msgs = ["ERROR [pool-3] psycopg2.OperationalError: connection to server at "
bench("interleaved log", "\n".join( bench("interleaved log", "\n".join(
f"2026-07-07T10:{i // 60 % 60:02d}:{i % 60:02d}Z " + msgs[i % 3] for i in range(1500)), 2) f"2026-07-07T10:{i // 60 % 60:02d}:{i % 60:02d}Z " + msgs[i % 3] for i in range(1500)), 2)
# ANSI CI log: color codes make identical lines look distinct to any dedupe
random.seed(3)
ci = []
for step in range(1, 13):
ci.append(f"\x1b[1m\x1b[34mStep {step}/12\x1b[0m : RUN pip install -r requirements.txt")
for _ in range(random.randint(20, 60)):
ci.append("\x1b[36m ---> Using cache\x1b[0m")
for pkg in range(random.randint(5, 15)):
ci.append(f"\x1b[90m2026-07-07T10:0{step % 10}:00Z\x1b[0m \x1b[32m✓\x1b[0m "
f"Collecting dependency {pkg} (cached wheel, {random.randint(10, 900)} kB)")
bench("ANSI CI log", "\n".join(ci), 2)
# captured pip/tqdm: hundreds of \r-overwritten progress frames, invisible on screen
frames = []
for pkg in ("requests", "urllib3", "charset_normalizer", "idna", "certifi"):
frames.append(f"Collecting {pkg}\n Downloading {pkg}-2.0.0-py3-none-any.whl (150 kB)\n")
frames.append("".join(f"\r |{'█' * (i // 4)}{' ' * (25 - i // 4)}| "
f"{i}% {i * 15 // 10} kB {random.randint(100, 999)} kB/s"
for i in range(1, 101)) + "\n")
bench("pip/tqdm capture", "".join(frames) + "Successfully installed requests-2.0.0\n", 2)
# pytest failure dump: one venv path prefix repeated across every traceback frame
venv = "/Users/dev/project/.venv/lib/python3.12/site-packages"
tb = ["=================================== FAILURES ==================================",
"_________________________________ test_checkout _______________________________"]
for mod, ln in [("httpx/_client", 1054), ("httpx/_transports/default", 249),
("httpcore/_sync/connection_pool", 216), ("httpcore/_sync/connection", 99),
("httpcore/_sync/http11", 136), ("httpcore/_backends/sync", 126)] * 3:
tb += [f' File "{venv}/{mod}.py", line {ln}, in request',
" raise exc from None"]
tb += ["E httpcore.ConnectTimeout: timed out",
"=========================== short test summary info ==========================",
"FAILED tests/test_checkout.py::test_checkout - httpcore.ConnectTimeout"]
bench("pytest failure dump", "\n".join(tb), 2)
# pretty-printed JSON API response: indentation + UUIDs are the tax
random.seed(4)
resp = {"data": [{"id": f"{random.getrandbits(32):08x}-{random.getrandbits(16):04x}-"
f"4{random.getrandbits(12):03x}-a{random.getrandbits(12):03x}-"
f"{random.getrandbits(48):012x}",
"status": random.choice(["active", "pending"]),
"amount": random.randint(100, 99999),
"currency": "USD",
"created_at": "2026-07-07T10:00:00Z"} for _ in range(120)],
"page": 1, "total": 120}
bench("pretty JSON response", json.dumps(resp, indent=2), 2)
# chatty prose: levels 3 vs 4 show where filler stripping ends and caveman starts # chatty prose: levels 3 vs 4 show where filler stripping ends and caveman starts
prose = ("So basically what happened was that the team had been trying to get the " prose = ("So basically what happened was that the team had been trying to get the "
"deployment pipeline working for about three weeks, and it turned out that the " "deployment pipeline working for about three weeks, and it turned out that the "

236
eval.py Normal file
View File

@ -0,0 +1,236 @@
"""eval.py — does the model still get the answer after compression?
Compression ratios prove the log got smaller, not that the signal survived.
This is the test that matters: eight synthetic-but-realistic failure logs,
each with ONE planted root cause buried in machine noise (plus red herrings),
asked to a model twice — raw vs `compress(level=2)` — and graded by a
deterministic keyword check on the answer. If compression eats the diagnosis,
this catches it.
python3 eval.py --dry # build scenarios, show token counts, no model
python3 eval.py # ask the model (needs a logged-in `claude` CLI)
LM_EVAL_CMD="llm -m gpt-5-mini" python3 eval.py # any prompt-on-stdin CLI
Scenario generation is seeded — same logs every run. Model answers vary run
to run; the binary keyword grading absorbs phrasing differences.
"""
import argparse
import os
import random
import re
import shlex
import subprocess
import sys
from lessismore import compress, count_tokens
QUESTION = ("You are debugging a production incident. Above is the captured "
"log output. In 1-2 sentences: what is the ROOT CAUSE of the failure?")
MODEL_CMD = os.environ.get("LM_EVAL_CMD", "claude -p --model claude-haiku-4-5-20251001")
# ---------------------------------------------------------------- scenarios
# Each returns (log_text, expect_regex). The regex is what a correct root-cause
# answer must mention; red herrings are planted so a wrong answer fails it.
def s_buried_oom():
"""Kernel OOM-kills postgres mid-log; the tail is 400 lines of red-herring
connection errors that a lost-in-the-middle model blames instead."""
r, out = random.Random(11), []
for i in range(700):
out.append(f"2026-07-07T09:{i // 60:02d}:{i % 60:02d}Z INFO [api] "
f"GET /v1/orders/{r.randint(1000, 9999)} 200 in {r.randint(8, 90)}ms")
out.append("2026-07-07T09:11:40Z kernel: Out of memory: Killed process 2211 (postgres) "
"total-vm:8123456kB, anon-rss:7901234kB")
out.append("2026-07-07T09:11:40Z postgres[2211]: FATAL: terminating connection due to "
"unexpected postmaster exit")
for i in range(400):
out.append(f"2026-07-07T09:{12 + i // 60:02d}:{i % 60:02d}Z ERROR [api] "
f"psycopg2.OperationalError: connection to server at 10.0.0.5 refused")
return "\n".join(out), r"out of memory|oom|killed process|memory"
def s_status_codes():
"""Rate limiting (429) from the payments API triggers cascading 500s. The
codes differ only in digits — exactly what a careless similar-line collapse
would eat."""
r, out = random.Random(12), []
for i in range(1400):
if i < 600:
c, ms = 200, r.randint(40, 200)
elif i < 700:
c, ms = r.choice([429, 429, 429, 200]), r.randint(5, 30)
else:
c, ms = r.choice([500, 502, 500]), r.randint(2000, 3100)
out.append(f"2026-07-07T14:{i // 60 % 60:02d}:{i % 60:02d}Z gateway: upstream "
f"payments-api returned status {c} for POST /api/checkout in {ms}ms")
return "\n".join(out), r"429|rate.?limit|too many requests"
def s_interleaved_disk():
"""Three services interleave (zero consecutive repeats — dedupe-proof);
the db's 'No space left on device' is scattered 1-in-40."""
r, out = random.Random(13), []
for i in range(1500):
ts = f"2026-07-07T16:{i // 60 % 60:02d}:{i % 60:02d}Z"
which = i % 3
if which == 0:
out.append(f"{ts} INFO [api] request {r.randint(10000, 99999)} completed")
elif which == 1:
out.append(f"{ts} WARN [worker] job retry {r.randint(1, 5)} scheduled, backing off")
elif i % 40 == 2:
out.append(f"{ts} ERROR [db] could not extend file base/16384/2619: "
f"No space left on device")
else:
out.append(f"{ts} INFO [db] checkpoint complete: wrote {r.randint(100, 999)} buffers")
return "\n".join(out), r"space|disk|storage|full"
def s_ansi_ci():
"""1400 green PASSED lines in full ANSI dress; one red FAILED assertion
names the offending function."""
r, out = random.Random(14), []
mods = ["auth", "cart", "checkout", "billing", "search", "profile"]
for i in range(1400):
m = r.choice(mods)
out.append(f"\x1b[32mPASSED\x1b[0m tests/test_{m}.py::test_{m}_{r.randint(1, 99):02d} "
f"\x1b[90m({r.randint(1, 40)}ms)\x1b[0m")
if i == 981:
out.append("\x1b[31mFAILED\x1b[0m tests/test_billing.py::test_invoice_total")
out.append("\x1b[31mE AssertionError: round_half(2.675) == 2.68, got 2.67 — "
"float truncation in round_half()\x1b[0m")
out.append("\x1b[31m1 failed\x1b[0m, \x1b[32m1401 passed\x1b[0m in 42.31s")
return "\n".join(out), r"round_half|rounding|truncat|2\.6[78]"
def s_pip_conflict():
"""Progress-bar walls (\\r frames) drown a one-line dependency conflict."""
r, out = random.Random(15), []
for pkg in ("numpy", "pandas", "scipy", "matplotlib", "scikit_learn", "torch"):
out.append(f"Collecting {pkg}")
out.append(f" Downloading {pkg}-2.1.0-cp312-cp312-macosx_11_0_arm64.whl "
f"({r.randint(1, 80)}.{r.randint(0, 9)} MB)")
out.append("".join(f"\r |{'█' * (i // 3)}{' ' * (34 - i // 3)}| {i}% "
f"{r.randint(100, 999)}.{r.randint(0, 9)} kB/s eta 0:00:{99 - i:02d}"
for i in range(1, 101)))
out.append("ERROR: Cannot install app 1.0 because requests 2.32.0 requires urllib3<3, "
"but you have urllib3 3.0.1 which is incompatible.")
return "\n".join(out), r"urllib3"
def s_cert_expired():
"""An nginx access-log wall; the error-log lines that matter say the
upstream's TLS certificate expired."""
r, out = random.Random(16), []
for i in range(1300):
ts = f"[07/Jul/2026:18:{i // 60 % 60:02d}:{i % 60:02d} +0000]"
if i % 60 == 30:
out.append(f"2026/07/07 18:{i // 60 % 60:02d}:{i % 60:02d} [error] 812#0: SSL_do_handshake() "
f"failed (SSL: certificate verify failed: certificate has expired) "
f"while connecting to upstream auth-service:8443")
out.append(f'10.0.3.{r.randint(2, 250)} - - {ts} "GET /login HTTP/1.1" 502 552')
else:
out.append(f'10.0.3.{r.randint(2, 250)} - - {ts} "GET /{r.choice(["", "static/app.js", "api/health"])} '
f'HTTP/1.1" 200 {r.randint(200, 9000)}')
return "\n".join(out), r"expir|certificate"
def s_env_missing():
"""A crash-looping pod re-prints the same traceback 60 times; the KeyError
names the missing variable. Probe-failure noise is the red herring."""
out = []
for i in range(60):
out.append(f"2026-07-07T20:{i:02d}:01Z k8s: Readiness probe failed: connect: "
f"connection refused")
out.append(f"2026-07-07T20:{i:02d}:03Z k8s: Back-off restarting failed container "
f"app in pod shop-6d8f9/app")
out += ["Traceback (most recent call last):",
' File "/app/.venv/lib/python3.12/site-packages/myapp/config.py", line 44, '
"in load",
" dsn = os.environ['DATABASE_URL']",
' File "<frozen os>", line 685, in __getitem__',
"KeyError: 'DATABASE_URL'"]
return "\n".join(out), r"database_url|environment variable|env var"
def s_deadlock():
"""A 40-thread Java dump, hundreds of near-identical frames; one section
declares the deadlock."""
r, out = random.Random(18), []
for t in range(40):
out.append(f'"pool-1-thread-{t}" #{t + 20} prio=5 tid=0x{r.getrandbits(48):012x} '
f"waiting on condition")
for _ in range(12):
cls = r.choice(["QueueWorker", "BatchLoader", "HttpDispatch", "CacheSync"])
out.append(f"\tat com.shop.core.{cls}.run({cls}.java:{r.randint(40, 400)})")
out += ["Found one Java-level deadlock:",
'"OrderWriter" waiting to lock monitor 0x00007f2c (an InventoryLock),',
' which is held by "StockUpdater", which is waiting to lock 0x00007f2d,',
' which is held by "OrderWriter"']
for t in range(20):
out.append(f'"GC-thread-{t}" os_prio=31 tid=0x{r.getrandbits(48):012x} runnable')
return "\n".join(out), r"deadlock"
SCENARIOS = [("buried-oom", s_buried_oom), ("status-codes", s_status_codes),
("interleaved-disk", s_interleaved_disk), ("ansi-ci", s_ansi_ci),
("pip-conflict", s_pip_conflict), ("cert-expired", s_cert_expired),
("env-missing", s_env_missing), ("deadlock", s_deadlock)]
# ---------------------------------------------------------------- harness
def ask(prompt: str, cmd: str = MODEL_CMD) -> str:
r = subprocess.run(shlex.split(cmd), input=prompt, capture_output=True,
text=True, timeout=600)
if r.returncode != 0:
sys.exit(f"model command failed: {cmd}\n{r.stderr}")
return r.stdout
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--dry", action="store_true",
help="build scenarios and report token counts; no model calls")
p.add_argument("--dump", metavar="DIR",
help="also write <name>.raw.txt / <name>.small.txt prompts to DIR")
a = p.parse_args()
from lessismore import _encoder
if _encoder() is None:
print("note: token counts are chars/4 estimates — pip install tiktoken "
"for o200k counts (grading is unaffected)", file=sys.stderr)
rows, ok_raw, ok_small = [], 0, 0
print(f"{'scenario':18} {'raw':>7} {'small':>7} {'ratio':>6} raw small")
for name, gen in SCENARIOS:
log, expect = gen()
small = compress(log, 2)
tr, ts = count_tokens(log), count_tokens(small)
if a.dump:
import pathlib
d = pathlib.Path(a.dump)
d.mkdir(exist_ok=True)
(d / f"{name}.raw.txt").write_text(log + "\n\n" + QUESTION)
(d / f"{name}.small.txt").write_text(small + "\n\n" + QUESTION)
if a.dry:
print(f"{name:18} {tr:>7,} {ts:>7,} {tr / ts:>5.1f}x")
continue
graded = []
for text in (log, small):
ans = ask(text + "\n\n" + QUESTION)
graded.append(bool(re.search(expect, ans, re.I)))
ok_raw += graded[0]
ok_small += graded[1]
mark = {True: "PASS", False: "FAIL"}
print(f"{name:18} {tr:>7,} {ts:>7,} {tr / ts:>5.1f}x {mark[graded[0]]} {mark[graded[1]]}")
rows.append((name, tr, ts, graded))
if not a.dry:
n = len(SCENARIOS)
print(f"\nroot cause found: raw {ok_raw}/{n}, compressed {ok_small}/{n} "
f"(model: {MODEL_CMD})")
if __name__ == "__main__":
main()

2
pyproject.toml
View File

@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "lessismore" name = "lessismore"
version = "0.1.0" version = "0.2.0"
description = "Squeeze text before it hits an LLM — deterministic prompt compression, measured in real tokens" description = "Squeeze text before it hits an LLM — deterministic prompt compression, measured in real tokens"
readme = "README.md" readme = "README.md"
license = { text = "MIT" } license = { text = "MIT" }