diff --git a/README.md b/README.md index ce44be2..ecee9e5 100644 --- a/README.md +++ b/README.md @@ -11,12 +11,16 @@ measurement is documented at the bottom, so you know exactly what you're getting. ```bash -tail -5000 app.log | python3 lessismore.py -l 2 | llm "why did this crash?" +tail -5000 app.log | python3 lessismore.py | llm "why did this crash?" ``` **11x** on mixed error logs · **12x** on interleaved service logs · -**8.9x** on ANSI CI logs · **98%** on captured pip/tqdm output · -**~6 MB/s** single-core · **0** dependencies +**11.8x** on ANSI CI logs · **97%** on captured pip/tqdm output · +**~6–12 MB/s** single-core · **0** dependencies + +And the part a compression ratio can't prove: in the root-cause eval +([eval.py](eval.py)), the model found the planted root cause in **8/8** +compressed logs vs **6/8** raw — smaller *and* more often right. ## Why this exists @@ -37,10 +41,15 @@ Three payoffs when you pipe through it: - **Context capacity** — an hour of log history fits where two minutes did. On a local model, that's the difference between full speed and crawling. -- **Prompt-cache longevity** — every pass is deterministic: same input, same - output, byte for byte. Follow-up questions re-hit the provider cache - instead of re-paying for the logs. An ML compressor in the loop would bust - the cache on every subtle variation; the regex passes never do. +- **Prompt-cache longevity** — every pass is deterministic and idempotent: + same input, same output, byte for byte (it's a test invariant). This + matters wherever a prompt is *re-generated* from source each call — a CI + assistant re-reading the same log, RAG context rebuilt per request, a hook + re-running on every tool call. A deterministic pass produces the identical + prefix every time, so the provider cache hits; an ML compressor in the + loop busts it on every subtle variation. (Within a single chat, follow-ups + hit the cache regardless — the transcript is append-only. The claim is + about rebuilt prompts, not chat turns.) - **Model attention** — LLMs lose things in the middle of walls of text. Feed signal, not noise, and the first answer is the right one more often. @@ -53,10 +62,12 @@ python3 test_lessismore.py # should print "ok" ``` ```bash -lm dump.log -l 2 > small.log # file in, file out -docker logs myapp 2>&1 | lm -l 2 # pipe filter -lm big.txt -l 2 --budget 4000 # compress, then hard-cap at ~4k tokens -lm big.txt -l 2 --ml 0.5 # + LLMLingua last mile +lm dump.log > small.log # file in, file out (level 2 is the default) +docker logs myapp 2>&1 | lm # pipe filter +lm --run "pytest -x" # run a noisy command, print it compressed, + # keep its exit code (pipes can't) +lm big.txt --budget 4000 # compress, then hard-cap at ~4k tokens +lm big.txt --ml 0.5 # + LLMLingua last mile lm --serve # paste-in demo page on localhost:7777 ``` @@ -77,10 +88,16 @@ Four levels, from byte-cautious to caveman. Pick by content, not by greed. | Level | What it eats | Point it at | Profile | |---|---|---|---| | **1** | whitespace runs, consecutive duplicate lines | code, scripts, anything | structure-safe: indentation and content untouched (whitespace *inside string literals* still collapses) | -| **2** | + JSON minify, `\r` redraw collapse, ANSI strip, ISO timestamps, base64/hex blobs, UUIDs, venv paths, scattered-duplicate aliasing, similar-line collapse | logs, CI dumps, traces, tool output | **the sweet spot** — destroys machine noise, keeps every distinct fact | +| **2** | + JSON minify (whole-input *and* embedded pretty blocks), `\r` redraw collapse, ANSI strip, ISO/syslog/nginx timestamps, base64/base64url/hex blobs, UUIDs, venv paths, scattered-duplicate aliasing, similar-line collapse with value summaries | logs, CI dumps, traces, tool output | **the sweet spot and the default** — destroys machine noise, keeps every distinct fact | | **3** | + filler-phrase stripping ("could you please", "just") | prose, chat history | fine for text, never for strict logic | | **4** | + `two_sticks` caveman mode: drops articles/copulas/auxiliaries, never negations or modals | gist-only prose, transcripts | lossy on style, protective of meaning — "do **not** delete" keeps its *not* | +"Keeps every distinct fact" is enforced, not hoped: when similar lines +collapse, the digits that varied are summarized in the marker — +`[3 similar lines omitted; values 404/429/503]` — because sometimes the +digits (status codes, ports, exit codes) *are* the diagnosis. An earlier +version silently ate them; the adversarial eval below is what caught it. + The crown jewel at level 2 is `alias_repeats`: ordinary dedupe only sees *consecutive* repeats, so interleaved multi-service logs sail straight through it. `alias_repeats` hunts scattered duplicates across the whole file and @@ -96,20 +113,95 @@ context limit is a hard wall. ## The receipts -All reproducible: `python3 bench.py` (o200k counts via tiktoken). +Every row reproducible from one command: `python3 bench.py` (o200k counts via +tiktoken — it refuses to print estimate numbers, so what you measure is what +this table claims). | Sample | Tokens | Why it wins | |---|---|---| | Mixed error log, 2000 lines | 86,914 → 7,895 (**11.0x**) | timestamp strip unmasks identical lines → dedupe; similar-line collapse catches the numbered stragglers | | Interleaved 3-service log, zero consecutive repeats | 54,999 → 4,585 (**12.0x**) | `alias_repeats` — consecutive dedupe alone managed 1.6x on this input | -| ANSI-colored CI/docker build log | 21,198 → 2,387 (**8.9x**) | color codes make identical lines look different; strip them and the log collapses | -| Captured pip/tqdm output | 14,483 → 291 (**98%**) | every overwritten `\r` progress frame is invisible on screen but real tokens in a capture | -| pytest failure dump | 2,223 → 1,673 (25%) | one site-packages path prefix = 25 tokens → 7 | -| Pretty-printed JSON API response | 11,741 → 6,833 (42%) | minify (round-trip verified lossless) + UUID→8-hex squash | +| ANSI-colored CI log | 10,440 → 888 (**11.8x**) | color codes make identical lines look different; strip them and the log collapses | +| Captured pip/tqdm output | 10,465 → 265 (**97%**) | every overwritten `\r` progress frame is invisible on screen but real tokens in a capture | +| pytest failure dump | 732 → 515 (30%) | one site-packages path prefix = 25 tokens → 7 | +| Pretty-printed JSON API response | 8,932 → 3,234 (**64%**) | minify (round-trip verified lossless) + UUID→8-hex squash | | Chatty prose, level 4 | 427 → 264 (38%) | every dropped function word is a whole token; level 3 got 2% on the same text | -Throughput: measured **~6 MB/s single-core** at level 2 (28 MB log in 4.4s). -No model, no network — break-even input size is effectively zero. +Throughput: measured **6–12 MB/s single-core** at level 2 (20 MB log in +1.7s). No model, no network — break-even input size is effectively zero. + +## Does the model still get the answer? + +A compression ratio proves the log got smaller, not that the signal +survived. So this is the eval that matters, and it's in the repo: +[eval.py](eval.py) generates eight seeded failure logs, each with one planted +root cause buried in machine noise **plus a red herring** (e.g. a cascade of +connection errors *caused by* an OOM kill 400 lines earlier), asks a model +"what's the root cause?" on the raw and the compressed version, and grades +the answer with a deterministic keyword check. + +Result (claude-haiku-4.5, 2026-07-07, one run; this run went through the +Claude Code agent harness with the same single-turn prompts — `eval.py`'s +default runner is a logged-in `claude -p`. Rerun it yourself): + +| Scenario | Raw tokens | Compressed | Raw | Compressed | +|---|---|---|---|---| +| OOM kill buried mid-log, connection-error red herring | 38,777 | 157 (**247x**) | ✗ blamed the red herring | ✓ found the OOM | +| 429 rate-limit burst hidden in a wall of status codes | 46,899 | 6,056 | ✗ "service overwhelmed" | ✓ named the 429s | +| Disk-full scattered through 3 interleaved services | 38,616 | 12,487 | ✓ | ✓ | +| One failed assertion in 1,400 ANSI test lines | 46,812 | 21,563 | ✓ | ✓ | +| Dependency conflict under progress-bar walls | 16,304 | 446 | ✓ | ✓ | +| Expired TLS cert in an nginx access-log wall | 56,696 | 34,474 | ✓ | ✓ | +| Missing env var in a crash-loop traceback | 7,680 | 1,610 | ✓ | ✓ | +| Java deadlock in a 40-thread dump | 8,157 | 8,041 (1.0x) | ✓ | ✓ | +| **Total** | **259,941** | **84,834 (3.1x)** | **6/8** | **8/8** | + +The two raw failures are the lost-in-the-middle effect this tool exists to +counter: the model latched onto the loud symptom cascade and never surfaced +the quiet cause. Compressed, the cause is impossible to miss. The honest +caveats: n=1 run per condition, synthetic logs authored by this project, one +model — which is exactly why the harness ships in the repo with fixed seeds. +`python3 eval.py --dry` costs nothing and shows the scenarios; point +`LM_EVAL_CMD` at any prompt-on-stdin CLI to grade your own model. + +## Where the tokens actually are: agent harnesses + +A human remembering to pipe is the demo. The 50k-token walls of 2026 enter +context windows as **tool output inside agent harnesses**, so lessismore +ships two integrations: + +**`lm --run`** — run the noisy command through the compressor and keep its +exit code (a shell pipe can't do that without pipefail games): + +```bash +lm --run "pytest -x" # agent sees 500 tokens, not 20,000 +lm --run "docker build ." +``` + +Tell your agent about it once (CLAUDE.md: *"run noisy commands through +`lm --run`"*) and every test run gets cheaper. + +**`lm --hook`** — a Claude Code `PostToolUse` hook that compresses Bash +output before it enters the model's context. Zero dependencies, no `jq`; +outputs under 2,000 chars pass through untouched. In +`.claude/settings.json`: + +```json +{ + "hooks": { + "PostToolUse": [ + { + "matcher": "Bash", + "hooks": [{ "type": "command", "command": "lm --hook" }] + } + ] + } +} +``` + +Because the passes are deterministic, hook-compressed transcripts stay +byte-stable across re-runs — the prompt-cache argument above is strongest +exactly here. ## What we refused to build (measured so it stays dead) @@ -143,7 +235,10 @@ db" → "was **not** the db"), URLs eaten as base64, crashes on empty and non-UTF-8 stdin, distinct SHA-256s falsely merging as "repeated", `two_sticks` eating "IT" and "US" as function words. Nine fixed with regression tests, four documented below, one wontfix (adversarial in-band -marker collision). `python3 test_lessismore.py` — 30 asserts, no framework. +marker collision). A second review round found the big one: similar-line +collapse silently eating distinct status codes — fixed with value summaries +and now guarded by the downstream eval. `python3 test_lessismore.py` — +42 asserts, no framework. ## Where it does nothing (on purpose) @@ -153,7 +248,7 @@ and machine noise; it cannot compress information, and doesn't pretend to. | Content | Expect | Verdict | |---|---|---| | Repetitive machine output | 5–12x, up to 50x on pathological repeats | the reason this exists | -| Structured data (JSON, tracebacks) | 25–42% | worthwhile, lossless where it fires | +| Structured data (JSON, tracebacks) | 30–64% | worthwhile, lossless where it fires | | Varied prose | ~2% (level 3) / 38% lossy (level 4) | gist mode only | | Clean code, unique dense text | ~0% by design | that's what `--ml` or truncation is for | @@ -162,6 +257,36 @@ last mile (`pip install llmlingua`) — runs *after* the deterministic passes so you're not paying a classifier model to delete duplicate log lines. Only worth it on multi-KB inputs, and it forfeits the cache-stability guarantee. +## Related work (know the river you're panning) + +Honest placement, so you can pick the right tool: + +- **[rtk](https://github.com/rtk-ai/rtk)** — Rust proxy with per-command + filters (ANSI strip, dedupe, failures-only test output), hooks into agent + CLIs. Command-aware and aggressive where lessismore is generic text passes + and lossless-leaning: rtk decides what you need to see; lessismore removes + only what is provably redundant and keeps a legend. +- **[Headroom](https://github.com/chopratejas/headroom)** — the heavyweight + tool-output compressor (Python, proxy + MCP). Has ML models in the loop, + so it needs a cache-alignment component to patch the non-determinism + lessismore doesn't have. +- **[Drain3](https://github.com/logpai/Drain3)** — industrial log template + mining. `alias_repeats` + `dedupe_similar` are a single-pass, zero-dep + approximation of it; if you need streaming template state across files, + use Drain3. +- **[LLMLingua-2](https://github.com/microsoft/LLMLingua)** — ML token + pruning; wrapped here as the optional `--ml` last mile, *after* the + deterministic passes, so you never pay a classifier to delete duplicate + log lines. +- **[Dictionary-encoding prompt compression](https://arxiv.org/abs/2604.13066)** + (2026) — independently validates the `@1 = line` legend technique + academically (≥0.99 fidelity on log benchmarks); no code released. + lessismore is, in effect, a reference implementation. + +What none of them package together — and the reason this exists — is the +combination: pure-stdlib, deterministic end to end, tokenizer-measured, with +the cache-stability argument as a design constraint rather than a patch. + ## Try it in a browser `lm --serve` runs a paste-in demo page on `http://localhost:7777` — paste your @@ -182,3 +307,9 @@ MIT — see [LICENSE](LICENSE). differently-phrased "retry the job" lines can legitimately merge. - In-band markers can collide with input that already contains them; `alias_repeats` bails out if its own `@N` markers already appear as lines. +- Blob squashing eats base64url runs (real JWTs) only when they look like + blobs (digits + mixed case); a 64-char kebab-case or snake_case identifier + is treated as content and survives. +- Similar-line collapse summarizes varying digits per column + (`values 404/429/503`, or a min–max range past 4 distinct); if the digit + *runs* per line differ in count, it falls back to a plain count marker. diff --git a/bench.py b/bench.py index cc4cbe6..6246c12 100644 --- a/bench.py +++ b/bench.py @@ -1,7 +1,19 @@ -"""Reproduce the README benchmark numbers. pip install tiktoken for exact counts.""" -import random +"""Reproduce the README benchmark numbers — every row of the receipts table. -from lessismore import compress, count_tokens +Requires tiktoken (pip install tiktoken): the README numbers are o200k counts, +and printing chars/4 estimates here would let the receipts silently drift from +the claims. Pass --estimate if you really want the guesses. +""" +import json +import random +import sys + +from lessismore import compress, count_tokens, _encoder + +if _encoder() is None and "--estimate" not in sys.argv: + sys.exit("bench.py: tiktoken not installed — these would be chars/4 GUESSES,\n" + "not the o200k-measured numbers the README claims.\n" + " pip install tiktoken (or pass --estimate to see the guesses anyway)") def bench(name, text, level): @@ -23,7 +35,6 @@ lines.insert(500, "2026-07-07T10:08:20.123Z DEBUG auth token=" + "eyJhbGciOiJIUz bench("mixed error log", "\n".join(lines), 2) # interleaved log: three services alternating — zero consecutive runs, dedupe-proof -random.seed(2) msgs = ["ERROR [pool-3] psycopg2.OperationalError: connection to server at " "10.0.0.5 failed: Connection refused", "WARN [redis-1] retry queue depth exceeded threshold, backing off 500ms", @@ -31,6 +42,53 @@ msgs = ["ERROR [pool-3] psycopg2.OperationalError: connection to server at " bench("interleaved log", "\n".join( f"2026-07-07T10:{i // 60 % 60:02d}:{i % 60:02d}Z " + msgs[i % 3] for i in range(1500)), 2) +# ANSI CI log: color codes make identical lines look distinct to any dedupe +random.seed(3) +ci = [] +for step in range(1, 13): + ci.append(f"\x1b[1m\x1b[34mStep {step}/12\x1b[0m : RUN pip install -r requirements.txt") + for _ in range(random.randint(20, 60)): + ci.append("\x1b[36m ---> Using cache\x1b[0m") + for pkg in range(random.randint(5, 15)): + ci.append(f"\x1b[90m2026-07-07T10:0{step % 10}:00Z\x1b[0m \x1b[32m✓\x1b[0m " + f"Collecting dependency {pkg} (cached wheel, {random.randint(10, 900)} kB)") +bench("ANSI CI log", "\n".join(ci), 2) + +# captured pip/tqdm: hundreds of \r-overwritten progress frames, invisible on screen +frames = [] +for pkg in ("requests", "urllib3", "charset_normalizer", "idna", "certifi"): + frames.append(f"Collecting {pkg}\n Downloading {pkg}-2.0.0-py3-none-any.whl (150 kB)\n") + frames.append("".join(f"\r |{'█' * (i // 4)}{' ' * (25 - i // 4)}| " + f"{i}% {i * 15 // 10} kB {random.randint(100, 999)} kB/s" + for i in range(1, 101)) + "\n") +bench("pip/tqdm capture", "".join(frames) + "Successfully installed requests-2.0.0\n", 2) + +# pytest failure dump: one venv path prefix repeated across every traceback frame +venv = "/Users/dev/project/.venv/lib/python3.12/site-packages" +tb = ["=================================== FAILURES ==================================", + "_________________________________ test_checkout _______________________________"] +for mod, ln in [("httpx/_client", 1054), ("httpx/_transports/default", 249), + ("httpcore/_sync/connection_pool", 216), ("httpcore/_sync/connection", 99), + ("httpcore/_sync/http11", 136), ("httpcore/_backends/sync", 126)] * 3: + tb += [f' File "{venv}/{mod}.py", line {ln}, in request', + " raise exc from None"] +tb += ["E httpcore.ConnectTimeout: timed out", + "=========================== short test summary info ==========================", + "FAILED tests/test_checkout.py::test_checkout - httpcore.ConnectTimeout"] +bench("pytest failure dump", "\n".join(tb), 2) + +# pretty-printed JSON API response: indentation + UUIDs are the tax +random.seed(4) +resp = {"data": [{"id": f"{random.getrandbits(32):08x}-{random.getrandbits(16):04x}-" + f"4{random.getrandbits(12):03x}-a{random.getrandbits(12):03x}-" + f"{random.getrandbits(48):012x}", + "status": random.choice(["active", "pending"]), + "amount": random.randint(100, 99999), + "currency": "USD", + "created_at": "2026-07-07T10:00:00Z"} for _ in range(120)], + "page": 1, "total": 120} +bench("pretty JSON response", json.dumps(resp, indent=2), 2) + # chatty prose: levels 3 vs 4 show where filler stripping ends and caveman starts prose = ("So basically what happened was that the team had been trying to get the " "deployment pipeline working for about three weeks, and it turned out that the " diff --git a/eval.py b/eval.py new file mode 100644 index 0000000..5f0157e --- /dev/null +++ b/eval.py @@ -0,0 +1,236 @@ +"""eval.py — does the model still get the answer after compression? + +Compression ratios prove the log got smaller, not that the signal survived. +This is the test that matters: eight synthetic-but-realistic failure logs, +each with ONE planted root cause buried in machine noise (plus red herrings), +asked to a model twice — raw vs `compress(level=2)` — and graded by a +deterministic keyword check on the answer. If compression eats the diagnosis, +this catches it. + + python3 eval.py --dry # build scenarios, show token counts, no model + python3 eval.py # ask the model (needs a logged-in `claude` CLI) + LM_EVAL_CMD="llm -m gpt-5-mini" python3 eval.py # any prompt-on-stdin CLI + +Scenario generation is seeded — same logs every run. Model answers vary run +to run; the binary keyword grading absorbs phrasing differences. +""" +import argparse +import os +import random +import re +import shlex +import subprocess +import sys + +from lessismore import compress, count_tokens + +QUESTION = ("You are debugging a production incident. Above is the captured " + "log output. In 1-2 sentences: what is the ROOT CAUSE of the failure?") + +MODEL_CMD = os.environ.get("LM_EVAL_CMD", "claude -p --model claude-haiku-4-5-20251001") + + +# ---------------------------------------------------------------- scenarios +# Each returns (log_text, expect_regex). The regex is what a correct root-cause +# answer must mention; red herrings are planted so a wrong answer fails it. + +def s_buried_oom(): + """Kernel OOM-kills postgres mid-log; the tail is 400 lines of red-herring + connection errors that a lost-in-the-middle model blames instead.""" + r, out = random.Random(11), [] + for i in range(700): + out.append(f"2026-07-07T09:{i // 60:02d}:{i % 60:02d}Z INFO [api] " + f"GET /v1/orders/{r.randint(1000, 9999)} 200 in {r.randint(8, 90)}ms") + out.append("2026-07-07T09:11:40Z kernel: Out of memory: Killed process 2211 (postgres) " + "total-vm:8123456kB, anon-rss:7901234kB") + out.append("2026-07-07T09:11:40Z postgres[2211]: FATAL: terminating connection due to " + "unexpected postmaster exit") + for i in range(400): + out.append(f"2026-07-07T09:{12 + i // 60:02d}:{i % 60:02d}Z ERROR [api] " + f"psycopg2.OperationalError: connection to server at 10.0.0.5 refused") + return "\n".join(out), r"out of memory|oom|killed process|memory" + + +def s_status_codes(): + """Rate limiting (429) from the payments API triggers cascading 500s. The + codes differ only in digits — exactly what a careless similar-line collapse + would eat.""" + r, out = random.Random(12), [] + for i in range(1400): + if i < 600: + c, ms = 200, r.randint(40, 200) + elif i < 700: + c, ms = r.choice([429, 429, 429, 200]), r.randint(5, 30) + else: + c, ms = r.choice([500, 502, 500]), r.randint(2000, 3100) + out.append(f"2026-07-07T14:{i // 60 % 60:02d}:{i % 60:02d}Z gateway: upstream " + f"payments-api returned status {c} for POST /api/checkout in {ms}ms") + return "\n".join(out), r"429|rate.?limit|too many requests" + + +def s_interleaved_disk(): + """Three services interleave (zero consecutive repeats — dedupe-proof); + the db's 'No space left on device' is scattered 1-in-40.""" + r, out = random.Random(13), [] + for i in range(1500): + ts = f"2026-07-07T16:{i // 60 % 60:02d}:{i % 60:02d}Z" + which = i % 3 + if which == 0: + out.append(f"{ts} INFO [api] request {r.randint(10000, 99999)} completed") + elif which == 1: + out.append(f"{ts} WARN [worker] job retry {r.randint(1, 5)} scheduled, backing off") + elif i % 40 == 2: + out.append(f"{ts} ERROR [db] could not extend file base/16384/2619: " + f"No space left on device") + else: + out.append(f"{ts} INFO [db] checkpoint complete: wrote {r.randint(100, 999)} buffers") + return "\n".join(out), r"space|disk|storage|full" + + +def s_ansi_ci(): + """1400 green PASSED lines in full ANSI dress; one red FAILED assertion + names the offending function.""" + r, out = random.Random(14), [] + mods = ["auth", "cart", "checkout", "billing", "search", "profile"] + for i in range(1400): + m = r.choice(mods) + out.append(f"\x1b[32mPASSED\x1b[0m tests/test_{m}.py::test_{m}_{r.randint(1, 99):02d} " + f"\x1b[90m({r.randint(1, 40)}ms)\x1b[0m") + if i == 981: + out.append("\x1b[31mFAILED\x1b[0m tests/test_billing.py::test_invoice_total") + out.append("\x1b[31mE AssertionError: round_half(2.675) == 2.68, got 2.67 — " + "float truncation in round_half()\x1b[0m") + out.append("\x1b[31m1 failed\x1b[0m, \x1b[32m1401 passed\x1b[0m in 42.31s") + return "\n".join(out), r"round_half|rounding|truncat|2\.6[78]" + + +def s_pip_conflict(): + """Progress-bar walls (\\r frames) drown a one-line dependency conflict.""" + r, out = random.Random(15), [] + for pkg in ("numpy", "pandas", "scipy", "matplotlib", "scikit_learn", "torch"): + out.append(f"Collecting {pkg}") + out.append(f" Downloading {pkg}-2.1.0-cp312-cp312-macosx_11_0_arm64.whl " + f"({r.randint(1, 80)}.{r.randint(0, 9)} MB)") + out.append("".join(f"\r |{'█' * (i // 3)}{' ' * (34 - i // 3)}| {i}% " + f"{r.randint(100, 999)}.{r.randint(0, 9)} kB/s eta 0:00:{99 - i:02d}" + for i in range(1, 101))) + out.append("ERROR: Cannot install app 1.0 because requests 2.32.0 requires urllib3<3, " + "but you have urllib3 3.0.1 which is incompatible.") + return "\n".join(out), r"urllib3" + + +def s_cert_expired(): + """An nginx access-log wall; the error-log lines that matter say the + upstream's TLS certificate expired.""" + r, out = random.Random(16), [] + for i in range(1300): + ts = f"[07/Jul/2026:18:{i // 60 % 60:02d}:{i % 60:02d} +0000]" + if i % 60 == 30: + out.append(f"2026/07/07 18:{i // 60 % 60:02d}:{i % 60:02d} [error] 812#0: SSL_do_handshake() " + f"failed (SSL: certificate verify failed: certificate has expired) " + f"while connecting to upstream auth-service:8443") + out.append(f'10.0.3.{r.randint(2, 250)} - - {ts} "GET /login HTTP/1.1" 502 552') + else: + out.append(f'10.0.3.{r.randint(2, 250)} - - {ts} "GET /{r.choice(["", "static/app.js", "api/health"])} ' + f'HTTP/1.1" 200 {r.randint(200, 9000)}') + return "\n".join(out), r"expir|certificate" + + +def s_env_missing(): + """A crash-looping pod re-prints the same traceback 60 times; the KeyError + names the missing variable. Probe-failure noise is the red herring.""" + out = [] + for i in range(60): + out.append(f"2026-07-07T20:{i:02d}:01Z k8s: Readiness probe failed: connect: " + f"connection refused") + out.append(f"2026-07-07T20:{i:02d}:03Z k8s: Back-off restarting failed container " + f"app in pod shop-6d8f9/app") + out += ["Traceback (most recent call last):", + ' File "/app/.venv/lib/python3.12/site-packages/myapp/config.py", line 44, ' + "in load", + " dsn = os.environ['DATABASE_URL']", + ' File "", line 685, in __getitem__', + "KeyError: 'DATABASE_URL'"] + return "\n".join(out), r"database_url|environment variable|env var" + + +def s_deadlock(): + """A 40-thread Java dump, hundreds of near-identical frames; one section + declares the deadlock.""" + r, out = random.Random(18), [] + for t in range(40): + out.append(f'"pool-1-thread-{t}" #{t + 20} prio=5 tid=0x{r.getrandbits(48):012x} ' + f"waiting on condition") + for _ in range(12): + cls = r.choice(["QueueWorker", "BatchLoader", "HttpDispatch", "CacheSync"]) + out.append(f"\tat com.shop.core.{cls}.run({cls}.java:{r.randint(40, 400)})") + out += ["Found one Java-level deadlock:", + '"OrderWriter" waiting to lock monitor 0x00007f2c (an InventoryLock),', + ' which is held by "StockUpdater", which is waiting to lock 0x00007f2d,', + ' which is held by "OrderWriter"'] + for t in range(20): + out.append(f'"GC-thread-{t}" os_prio=31 tid=0x{r.getrandbits(48):012x} runnable') + return "\n".join(out), r"deadlock" + + +SCENARIOS = [("buried-oom", s_buried_oom), ("status-codes", s_status_codes), + ("interleaved-disk", s_interleaved_disk), ("ansi-ci", s_ansi_ci), + ("pip-conflict", s_pip_conflict), ("cert-expired", s_cert_expired), + ("env-missing", s_env_missing), ("deadlock", s_deadlock)] + + +# ---------------------------------------------------------------- harness + +def ask(prompt: str, cmd: str = MODEL_CMD) -> str: + r = subprocess.run(shlex.split(cmd), input=prompt, capture_output=True, + text=True, timeout=600) + if r.returncode != 0: + sys.exit(f"model command failed: {cmd}\n{r.stderr}") + return r.stdout + + +def main(): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("--dry", action="store_true", + help="build scenarios and report token counts; no model calls") + p.add_argument("--dump", metavar="DIR", + help="also write .raw.txt / .small.txt prompts to DIR") + a = p.parse_args() + + from lessismore import _encoder + if _encoder() is None: + print("note: token counts are chars/4 estimates — pip install tiktoken " + "for o200k counts (grading is unaffected)", file=sys.stderr) + + rows, ok_raw, ok_small = [], 0, 0 + print(f"{'scenario':18} {'raw':>7} {'small':>7} {'ratio':>6} raw small") + for name, gen in SCENARIOS: + log, expect = gen() + small = compress(log, 2) + tr, ts = count_tokens(log), count_tokens(small) + if a.dump: + import pathlib + d = pathlib.Path(a.dump) + d.mkdir(exist_ok=True) + (d / f"{name}.raw.txt").write_text(log + "\n\n" + QUESTION) + (d / f"{name}.small.txt").write_text(small + "\n\n" + QUESTION) + if a.dry: + print(f"{name:18} {tr:>7,} {ts:>7,} {tr / ts:>5.1f}x") + continue + graded = [] + for text in (log, small): + ans = ask(text + "\n\n" + QUESTION) + graded.append(bool(re.search(expect, ans, re.I))) + ok_raw += graded[0] + ok_small += graded[1] + mark = {True: "PASS", False: "FAIL"} + print(f"{name:18} {tr:>7,} {ts:>7,} {tr / ts:>5.1f}x {mark[graded[0]]} {mark[graded[1]]}") + rows.append((name, tr, ts, graded)) + if not a.dry: + n = len(SCENARIOS) + print(f"\nroot cause found: raw {ok_raw}/{n}, compressed {ok_small}/{n} " + f"(model: {MODEL_CMD})") + + +if __name__ == "__main__": + main() diff --git a/pyproject.toml b/pyproject.toml index 76eacc4..974f4ff 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "lessismore" -version = "0.1.0" +version = "0.2.0" description = "Squeeze text before it hits an LLM — deterministic prompt compression, measured in real tokens" readme = "README.md" license = { text = "MIT" }