bitmax/docs/handbook/11-vol-II-defrag.md

Vol II Internals — DEFRAG.EXE

A maintainer's tour of defrag.html: the cluster grid, the head sweep, the entropy engine that almost killed the volume, and the equilibrium that saved it.

This chapter is the engineering bible for Vol II — DEFRAG.EXE (defrag.html, save key boringsoft_defrag_v1). Everything here is drawn from the source; the identifiers, constants, and formulas are real. The hook — you steer a defrag — is a spatial simulation laid over the shared scaffolding from index.html (see Shared Technical DNA), so this chapter spends most of its words on the parts that are genuinely new: the grid, the sweep, the rot, and the balance hack that keeps the rot survivable.

The cell model

The entire playfield is one flat Int8Array named grid, indexed in reading order (i = y*cols + x). Six cell-state constants are packed in at the top of the script:

/* cell state codes (flat Int8Array) */
const FREE=0, FRAG=1, CONT=2, YOU=3, SYS=4, BAD=5;

Const	Value	Color (COLORS[])	Meaning	Movable?
FREE	0	#ffffff white	unused cluster — a relocation target	—
FRAG	1	#d83a2a red	scattered data; what the head hunts	yes
CONT	2	#1038b0 blue	consolidated data; pays passive	yes (but packed)
YOU	3	#18c8d8 cyan	FRAG's own consciousness fragments	yes
SYS	4	#e8c020 yellow	boot sector / MFT — unmovable	no
BAD	5	#101010 black	corruption; spreads	no

There are no per-cell objects. A cell is its Int8Array slot, which is what makes a 96×52 = 4,992-cluster grid cheap to simulate and repaint every tick. Two parallel typed arrays carry the rest of the per-cell state that doesn't fit in one byte: badAge (a Float64Array of Date.now()-relative spread deadlines) and the head/queue structures, which live outside the grid entirely.

The grid dimensions live in module-scope cols/rows, seeded to 40/24 but overwritten on every build and load. Always read them from the globals; never assume the starting size.

buildGrid()

buildGrid() is the deterministic-shape, random-fill generator. It pulls cols/rows/value/decay from the current drive tier, allocates new Int8Array(cols*rows).fill(FREE), then layers content in a fixed order:

const sysCount=Math.floor(total*0.05);          // 5% SYS band, top-left
const dataCount=Math.floor(total*0.55);          // 55% FRAG, scattered
let youCount=Math.max(4, Math.floor(total*0.012)); // ~1.2% YOU
let badCount=Math.max(1, Math.floor(total*0.004)); // ~0.4% starting BAD

SYS is written as a contiguous run from index 0 — a yellow band in the top-left, exactly where a real boot sector and master file table would sit. Everything else is scattered with a rejection-sampling loop bounded by a guard counter (guard++ < total*4), so a near-full grid can never spin forever. YOU cells are only ever placed onto existing FRAG, so your consciousness starts embedded in the noise. After the fill, the Compression Seed perk runs preConsolidate(seedPct), then badQueue is cleared and initBadAge() / initHeads() seed the auxiliary arrays.

DEV NOTE — the guard ceilings are not paranoia. The seed perk (pseed) and a high drive tier can leave very few FREE/FRAG cells; without the guards, the placement while loops would hang the boot thread on a starved grid. The cost of an occasional under-fill (a few clusters short of target) is invisible; a hung tab is not. Trade accepted.

The grid ↔ G.grid sync

grid is the live typed array; G.grid is its serialized cousin for localStorage. They are deliberately not the same object. The bridge is two functions.

On save, serialize() snapshots the live array into the state object (defrag.html:1443):

G.grid=grid?Array.from(grid):null;
G.heads=heads.map(h=>({x:h.x,y:h.y,focus:h.focus||null}));
G.badQueue=badQueue.slice();

On load, rebuildFromSave() reverses it — but only if the shape still matches the current tier (defrag.html:1481):

if(Array.isArray(G.grid) && G.grid.length===want){
  grid=new Int8Array(want);
  for(let i=0;i<want;i++){ const v=G.grid[i]; grid[i]=(v>=FREE&&v<=BAD)?v:FREE; }
  initBadAge();
} else {
  buildGrid();
}

The want = cols*rows length check is load-bearing: if a save predates a balance change to the tier table, or is otherwise the wrong size, the restore silently falls back to a fresh buildGrid() rather than reading off the end of a too-short array. The per-cell clamp (v>=FREE&&v<=BAD)?v:FREE is the second guard — any garbage byte becomes harmless FREE. Note that badAge is never persisted; it is regenerated from scratch by initBadAge(), so a reload resets every bad sector's spread clock. That is intentional — it's one byte of mercy on every return.

· · ·

The head sweep

heads is an array of {x, y, focus} cursors. The simulation walks each one every tick inside step(dt).

The per-head budget is throughput in clusters, computed fresh each tick:

const perHead = headSpeed()*cacheWidth()*dt;

headSpeed() is BASE_HEAD * Math.pow(1.5, lvl('speed')) with BASE_HEAD = 4 clusters/sec at level 0; cacheWidth() is 1 + lvl('cache'). A fractional _carry accumulator per head banks sub-cluster budget between ticks so slow early-game heads still make progress. When a head holds a focus region (from the hook), its budget is multiplied by burstPower()*1.4 and its mint multiplier becomes burstPower().

stepHead(h, budget, mintMult) is the inner loop. It scans forward in reading order via advanceHead(), wrapping at the region or grid edge, until it finds a FRAG (or an unpacked YOU), then calls consolidateAt(). The relocation is the heart of the whole game:

function consolidateAt(idx, intoLeft, mintMult){
  const wasYou = grid[idx]===YOU;
  let target=idx;
  if(intoLeft){
    const f=firstFreeBefore(idx);
    if(f>=0 && f<idx){ target=f; grid[idx]=FREE; }
  }
  grid[target]= wasYou?YOU:CONT;
  const pay = clusterValue() * (mintMult||1);
  G.rb += pay; G.totalRb += pay; G.runReclaimed += 1;
  return pay;
}

firstFreeBefore(idx) finds the leftmost FREE slot before idx, so data is packed toward index 0 — visibly draining red toward a clean blue band on the left. Each consolidation mints clusterValue() Reclaimed Bytes and increments runReclaimed (the platter-calculation counter, see Prestige). The isPacked() predicate stops the head from endlessly re-relocating a YOU cell that already has no free space to its left.

clusterValue, algoMult, globalMult

The per-cluster payout is a product of three terms, each a clean power-law:

Function	Formula	Driver
clusterValue()	driveDef(tier).value * algoMult() * globalMult()	the whole stack
algoMult()	Math.pow(2.2, lvl('algo'))	Consolidation Algorithm upgrade
globalMult()	Math.pow(1.6, lvl('compact')) * (1 + 0.3*plvl('pmult'))	Aggressive Compaction + Cold Storage perk

globalMult() is the one term that folds in a persistent perk (pmult), which is why it appears in both the active mint and the passive-income path — it must apply to every byte the drive ever earns. The base value comes straight from the tier table and spans 64 (40 MB) to 1.3e8 (1 PB), so the headline number climbs through SI suffixes purely from the hardware ladder.

The automation layer

Two one-time upgrades steer heads without the player. The Defragmentation Daemon (daemon) gently biases an idle head's y toward fragHotspot() — the row with the most FRAG — but only with a 4% per-tick probability, so the steer feels like drift, not teleportation. The Auto-Repair Daemon (autorepair) is covered under Repair, below.

· · ·

Passive income from contiguous runs

Beyond per-cluster mints, contiguous blue pays rent. passiveRate() scans each row, accumulates run lengths of CONT/YOU, and pays each run through runPay():

function runPay(run){
  return run * driveDef(G.driveTier).value * 0.06 * (1 + run*0.04) * algoMult();
}

The (1 + run*0.04) term is super-linear: a run of 20 contiguous clusters earns more per cluster than two runs of 10. That is the entire design argument for packing left into long unbroken bands rather than scattering consolidations. passiveRate() then multiplies the row total by streakMult() (1 + 0.35*lvl('streak')) and globalMult(), and step() banks passiveRate()*dt into both rb and totalRb every tick.

Two derived gauges read the same grid:

function consolidatedPct(s){
  const movable=s.frag+s.cont+s.you;
  if(movable<=0) return 100;
  return clamp((s.cont+s.you)/movable*100, 0, 100);
}
function diskHealth(s){
  return clamp(100 - s.bad/totalCells()*100, 0, 100);
}

consolidatedPct() is the progress bar and the prestige gate — it counts only movable cells, so SYS and BAD don't poison the denominator. diskHealth() is the SMART gauge, a pure function of bad-sector fraction. When health drops to ≤0.5%, step() sets G._readonly and the head stops consolidating — the soft cap that pushes a dying drive toward migration.

DEV NOTE — consolidatedPct() returning 100 when movable<=0 is not a rounding accident. A grid that has been fully consolidated (or pre-seeded into oblivion) has zero movable cells, and dividing by zero there would paint NaN% on the progress bar and break the migrate gate. "Nothing left to move" is, correctly, "100% done."

· · ·

Entropy — the engine that almost shipped a death spiral

Entropy is the antagonist, and it is the one system that was redesigned after a post-mortem. The driver is driveDecay():

function driveDecay(){ return driveDef(G.driveTier).decay * entropyMult(); }

entropyMult() is Math.pow(0.91, lvl('cooling')) * Math.pow(0.92, plvl('pcool')) — the only thing the player can do to slow the clock. entropyTick(dt) accumulates real time and fires rollEntropy() once per period = 2.4 / driveDecay() seconds, so a higher-tier drive (decay up to 6.0) rolls the dice far more often.

rollEntropy() is an 80/20 split:

function rollEntropy(){
  const r=Math.random();
  if(r<0.80){
    const i=findRefragTarget();
    if(i>=0){ if(grid[i]===CONT) grid[i]=FRAG; else if(grid[i]===FREE) grid[i]=FRAG; }
  } else {
    if(badNow() < badCap()){
      const i=findBadTarget();
      if(i>=0){ grid[i]=BAD; badAge[i]=now()+rand(5000,10000)/driveDecay(); }
    }
  }
}

80% of rolls re-fragment — findRefragTarget() prefers flipping a CONT cell back to FRAG (visible decay of your work) and falls back to spawning fresh FRAG in free space. 20% spawn a bad sector, and — critically — only if badNow() < badCap(). Separately, spreadBad() runs every tick: each BAD cell whose badAge deadline has passed infects one random orthogonal neighbor (skipping SYS and other BAD), then resets its own deadline. Queued sectors are frozen — if(badQueue.indexOf(i)>=0) continue; — so corruption you've already targeted can't metastasize while it waits.

badCap() / badNow() — the equilibrium

Here is the fix, and the reason the comment in the source reads like a confession:

// Entropy reaches an EQUILIBRIUM instead of devouring the whole disk while you idle —
// bad sectors never grow past this cap passively (you repair to push below it). Bigger,
// older drives rot a little more, but it is never lethal.
function badCap(){ return Math.floor(totalCells() * Math.min(0.34, 0.18 + 0.02*G.driveTier)); }
function badNow(){ let b=0; for(let i=0;i<grid.length;i++) if(grid[i]===BAD) b++; return b; }

badCap() is a hard ceiling on passive corruption: 18% of the grid at tier 0, climbing 2 points per tier, clamped at 34%. Both entropy paths — the rollEntropy() spawn and the spreadBad() infection — check badNow() < badCap() before adding a black cell. The player's repair charges are the only thing that can push bad sectors below the cap; entropy can never push them above it on its own.

This exists because the first cut didn't have it. Without a cap, spreadBad() is exponential — bad sectors spawn bad sectors — and an idle drive would wake up at 100% black, health zero, read-only, unrecoverable. That violates Design Law 5: idle must reach equilibrium, never death. The cap converts a runaway feedback loop into a stable setpoint. You can walk away from a defrag and come back to a drive that's a little rotten and entirely playable. The min(0.34, …) clamp guarantees that even the 1 PB array, decaying at 6.0, tops out at roughly a third corrupted — annoying, never terminal.

DEV NOTE — the death spiral shipped, briefly, in a build with no cap and spreadBad() running on every tick. An eight-hour offline session would return a solid black grid and a save you couldn't dig out of: every repair you bought was immediately re-corrupted faster than you could spend charges. The lesson became canon. A cap is cheaper than a comeback mechanic, and equilibrium is a feature you can feel even when you can't see it. While paused, entropy still runs — entropyTick(G.running?dt:dt*0.4) — at 40% rate, so pausing slows the rot but never stops it. The tension is always on.

· · ·

Repair — charges, queue, daemon

Repair is the player's lever against BAD. maxCharges() is BASE_CHARGES + lvl('ecc') + 2*plvl('pcharge') (base 3), and charges regenerate one at a time over regenTime() seconds (REGEN_BASE/Math.pow(1.4, lvl('regen'))/(1+0.25*plvl('pcharge')), base 8s).

Clicking a black cell calls queueRepair(idx), which spends a charge and pushes the index onto badQueue. processRepairs(dt) works the queue head-first: a queued sector accrues G._repairProg and flips to FREE after 0.6s. The Auto-Repair Daemon automates targeting:

function autoRepair(){
  if(!has('autorepair')) return;
  if(G.charges<1) return;
  if(badQueue.length>=2) return;
  // …find nearest BAD to any head by Manhattan distance, then queueRepair(best)
}

It deliberately caps its own queue at 2 and bails when out of charges, so it assists rather than trivializes containment.

· · ·

The hook — drag-select and click

The mouse handlers live in one block. mousedown records downCell and starts a dragSel rubber-band; mousemove extends it and flips dragStarted once the box exceeds one cell; mouseup decides intent by area:

if(dragStarted && area>1){
  focusBurst(r);
} else if(downCell){
  clickCell(downCell);
}

focusBurst(r) assigns the normalized region as a focus to the one or two nearest heads (two if the region holds >40 FRAG, for a satisfying swoop) and snaps them to its top-left corner. A focused head gets the burstPower() mint and burstPower()*1.4 speed bonus from step(). clickCell(c) branches on the cell: a BAD cell queues for repair; anything else builds a small 5×3 focus box around the click and steers the nearest head there for a micro-defrag. cellFromEvent() maps screen pixels to grid coordinates using the same ox/oy/cellPx the renderer computes, so clicks land where they look.

· · ·

The drive ladder

DRIVES is the prestige tier table. driveDef(i) clamps the index to the array bounds. Bigger drives pay more and rot faster — the central tension of every migration.

Tier	Label	name	cols × rows	clusters	value	decay
0	40 MB	C:	40 × 24	960	64	1.00
1	540 MB	D:	48 × 28	1,344	512	1.35
2	6 GB	E:	56 × 32	1,792	4,096	1.75
3	40 GB	F:	64 × 36	2,304	3.3e4	2.25
4	120 GB	G:	72 × 40	2,880	2.6e5	2.9
5	2 TB	H:	80 × 44	3,520	2.1e6	3.7
6	16 TB	S:	88 × 48	4,224	1.7e7	4.7
7	1 PB Array	Z:	96 × 52	4,992	1.3e8	6.0

Migration

The prestige gate is PRESTIGE_PCT = 60 percent consolidated and at least one Platter earned. The payout is sub-linear in clusters reclaimed:

function plattersFor(){
  const def=driveDef(G.driveTier);
  const reclaimedFrac = Math.min(1, G.peakConsolidated/100);
  const base = Math.sqrt(G.runReclaimed/40) * (1+G.driveTier*0.6) * reclaimedFrac;
  return Math.floor(base);
}

The Math.sqrt(runReclaimed/40) term means Platters scale with the square root of effort — diminishing returns within a run, which is what pushes you to migrate up rather than farm one drive forever. doMigrate() banks the Platters, increments driveTier (capped at the last tier), wipes the run (rb, runReclaimed, peakConsolidated, bestRun, and critically G.upgrades={}), preserves perks, then calls buildGrid() on the new, larger platter. Perks persist; hardware upgrades do not. That is the reincarnation contract.

· · ·

Upgrades and perks

Upgrades are bought with Reclaimed Bytes in the "Defragmentation Options" dialog and reset on migration. Cost is base * growth^level for leveled upgrades, flat for one-time installs.

id	Name	tab	kind	base	growth	max	Effect
speed	Head Speed	head	lvl	48	1.55	60	head throughput ×1.5/lvl
cache	Read-Ahead Cache	head	lvl	120	1.7	24	sweep width 1+lvl
heads	Multiple Heads	head	lvl	800	2.15	11	+1 parallel head/lvl
algo	Consolidation Algorithm	head	lvl	300	2.4	7	$/cluster ×2.2/lvl
streak	Contiguity Bonus	head	lvl	420	1.95	20	streak income +35%/lvl
ecc	ECC / Repair Passes	repair	lvl	260	1.85	16	max charges +1/lvl
regen	Repair Regen Rate	repair	lvl	340	1.8	20	regen ×1.4/lvl
autorepair	Auto-Repair Daemon	repair	once	9000	—	—	auto-targets nearest BAD
cooling	Cooling / Anti-Entropy	repair	lvl	500	1.9	18	entropy ×0.91/lvl
daemon	Defragmentation Daemon	auto	once	6000	—	—	idle heads steer to hotspot
prefetch	Smart Prefetch	auto	lvl	1500	2.0	12	burst power +50%/lvl
compact	Aggressive Compaction	auto	lvl	2200	2.1	14	all income ×1.6/lvl

The algo upgrade also advances the cosmetic ALGO_NAMES ladder: BubbleSort → InsertionSort → QuickSort → HeapSort → RadixSort → TimSort → "I wrote my own" → Singularity-Sort.

Perks are bought with Platters in the migration wizard and persist across every drive forever.

id	Name	base	growth	max	Effect
pmult	Cold Storage Dividend	1	1.7	40	+30% all income/lvl
pcool	Cryogenic Platters	1	1.8	30	base entropy ×0.92/lvl
pheads	Spare Head Assembly	2	2.0	10	+1 starting head/lvl
pcharge	Hardened Sectors	2	1.9	12	+2 charges & faster regen/lvl
pseed	Compression Seed	3	2.2	10	8% pre-consolidated/lvl

· · ·

Rendering

The grid is a single <canvas>, repainted whole every tick by drawGrid() — not 4,992 DOM nodes. resizeCanvas() computes a square cellPx that fits the frame and centers the grid with ox/oy offsets, capping device-pixel-ratio at 2 to keep large drives cheap. The draw order is: white background, then every cell as a cellPx-1 square (the -1 leaves a one-pixel grout line for the grid texture), then YOU glow rings, then orange outlines on badQueue entries, then green head highlights with their faint vertical guides, then the dashed navy rubber-band if a drag is live. image-rendering: pixelated on the canvas keeps the cells crisp.

The HUD is plain DOM, throttled to repaint at most every 0.2s (railAccum). renderRail(), renderProgress(), and renderStatus() read the cached lastRailStates snapshot rather than rescanning the grid on every call. The status line cycles through a 2.5-second statusFlash override, then read-only / paused warnings, then the live "Compacting cluster N…" message — the same diegetic chatter a real defragmenter prints.

If you change this

• The entropy/bad-cap balance is the load-bearing wall. If you touch badCap(), rollEntropy(), or spreadBad(), re-derive the worst case: simulate an 8-hour offline session (MAX_OFFLINE_S) on the top tier (decay: 6.0) with zero cooling/pcool. The grid must come back below 34% bad and above read-only. Both spawn paths must keep their badNow() < badCap() guard, or the death spiral returns. This is not a knob; it is the contract.
• DRIVES tier shapes feed the save sync. Changing any cols/rows changes cols*rows, which is the exact length rebuildFromSave() checks against G.grid.length. Existing saves on a resized tier will silently regenerate via buildGrid() — acceptable, but know it happens. Never reorder or remove tiers without a save-version bump.
• value and decay move together. The whole feel is "bigger pays more, rots faster." If you raise a tier's value without raising decay, you flatten the difficulty curve; the migration decision stops being a trade-off.
• badAge is not persisted. If you ever need bad-sector spread timing to survive reload, you must add it to serialize()/rebuildFromSave() and re-clamp it in sanitizeState(). Today it's regenerated by initBadAge() on purpose — don't half-migrate it.
• globalMult() is in two hot paths. It's folded into both clusterValue() (active mint) and passiveRate() (streak income). Add a new global multiplier to globalMult() itself, never to one call site, or active and passive income will drift apart and the numbers stop being legible (Design Law 4).
• consolidatedPct() counts only movable cells. If you change what "consolidated" means, check the prestige gate (PRESTIGE_PCT), the progress bar, and the movable<=0 → 100 early return together. They all read this one function.