Lane E R18 (v4.0-alpha): katoomba_real real road geometry (schema v2, ledger #3)

Extend build_towns.py: bounded way[highway](bbox) out geom per town -> raw cached in-repo -> roads[]
in A's v2 contract ({id,kind,pts:[[lat,lon]]}, kind=raw OSM highway class). katoomba_real is now
schema v2 with 484 real streets (Bathurst Rd, Cliff Drive, ...). Fidelity cut (charter risk #4, flagged
to A): fetch every ROAD_KIND tier -> raw; EMIT the town street tier only, excluding service/track/
footway/path/steps/cycleway (1461 of 2506 Katoomba ways = bushwalking tracks/sidewalks/stairs/driveways).

A's lift builds a real town from it (verified): 484 streets -> 872 nodes/799 edges (main 232 side 531
arcade 36) -> 12 blocks, 20 lots, 3 venues (pub/band_room/rsl), 12 gigs. pickVenues lands venues on the
real graph (ledger #2 met).

FINDING for A (alpha failure-list, charter risk #3): on the real graph, spine posters overhang crossing
carriageways -- selfcheck kerb-clearance fails on 4 seeds (-1.6 to -2.5m; e.g. seed 20261990 poster 50 vs
edge 58 = -2.42m). Same class as A's R15/R16 corner-poster fix, now on real cross-street geometry. Owner:
A (pickVenues/POSTER_CLEAR on the real graph, the 'A again' harden step) -- my roads data is VALID (cache
validates; failures are plan-level poster placement). Golden settles after A's fix (pre-fix 0x6014a8e6).
ODbL: raw roads committed, SOURCES.md updated.

-> Lane A: katoomba roads live; harden the lift + resolve the poster finding + re-pin the golden.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/LANES/LANE_E_NOTES.md

@@ -1,5 +1,44 @@
 # LANE E — cross-lane notes
 
+## Round 18 — v4.0-alpha REAL ROADS: the road-geometry fetch ⟨v4-alpha⟩
+
+> Ledger #3 — E fetches the real street geometry; A's lift builds the CityPlan graph from it.
+> **`katoomba_real` is now schema v2 with 484 real streets** (the alpha town). → **Lane A: roads are
+> live; harden the lift, resolve the surfaced poster-clearance finding below, re-pin the golden.**
+
+**The data.** `build_towns.py` extended with a bounded `way["highway"~…](bbox); out geom;` per town →
+raw cached in `_raw/<key>.roads.overpass.json` (fetch once, John's standing Overpass auth) → `roads[]`
+in A's v2 contract (`{id, kind, pts:[[lat,lon],…]}`; `kind` = raw OSM highway class, A's `ROAD_KIND`
+maps it, full geometry shipped for A's Douglas–Peucker). **katoomba_real: 484 streets** (residential
+243, tertiary 65, trunk 58, secondary 51, primary 32, unclassified 23, pedestrian 12 — real names:
+Bathurst Rd, Cliff Drive, Civic Place, …), 5344 points.
+
+**Fidelity cut (charter risk #4 — flagged to A, who owns the knob).** I FETCH every `ROAD_KIND`-mapped
+tier (→ raw cache, full provenance / beta-ready) but EMIT only the **town street tier** — drivable
+roads + pedestrian malls — EXCLUDING `service`/`track`/`footway`/`path`/`steps`/`cycleway`. In
+Katoomba's 3 km bbox that excluded tier is **1461 of 2506 ways** (Blue-Mountains bushwalking tracks,
+sidewalks, stairs, driveways, parking aisles) — noise for a town street graph and a ~2500-edge blow-up.
+The full set is in the raw cache if A wants a tier back (one filter change). This is the honest cut.
+
+**A's lift builds a real town from it** (verified, `generatePlanFor` seed 20261990): 484 streets →
+**872 nodes / 799 edges** (main 232, side 531, arcade 36) → 12 blocks, 20 lots, 20 shops, **3 venues
+(pub/band_room/rsl), 12 gigs**. `pickVenues` lands venues on the real graph — ledger #2 requirement met.
+
+**⚠ FINDING for Lane A (the alpha's failure-list deliverable — charter risk #3).** On the real graph,
+**spine posters overhang a crossing carriageway**: selfcheck's kerb-clearance gate fails on 4 seeds
+(spine poster clears its cross-edge by **−1.6 to −2.5 m**; e.g. seed 20261990 poster 50 vs edge 58 =
+−2.42 m). Same CLASS as A's R15/R16 corner-poster fix, now on **real** cross-street geometry — a
+venue's frontage seats on the spine where a real side street crosses, so its poster reaches over that
+carriageway. **Owner: A** (`pickVenues` / `POSTER_CLEAR` on the real graph — the "A again" harden
+step). **My roads data is valid** — the v2 cache validates (roads accepted); the 4 failures are
+plan-level poster placement, not cache errors. Not a fidelity artifact either: the crossing streets are
+real primary town streets, so trimming roads wouldn't fix it. Golden settles after A's fix (pre-fix
+value `0x6014a8e6`). Handshake E→A: **katoomba roads live + this finding**.
+
+**Other 4 towns:** roads produced + committed as data-only (A hardens against them as they land; only
+katoomba gates the round). Roads are ODbL (© OSM contributors); raw responses committed, `SOURCES.md`
+updated with the roads tier + the fidelity cut.
+
 ## Round 17 — v3.2 THE SCOUT: real town caches + the verify fix ⟨v3.2 amendment⟩
 
 > The real-map scout for **v4 THE REAL MAP** (John's pick) — bounded data groundwork, no new game

pipeline/build_towns.py

@@ -43,6 +43,18 @@ TYPE_MAP = {
     "charity": "opshop", "second_hand": "opshop", "antiques": "opshop",
 }
 
+# ── roads[] (schema v2, ROUND18 REAL ROADS) ─────────────────────────────────────────────────────
+# We FETCH every OSM highway tier A's ROAD_KIND maps (→ raw cache, full provenance / beta-ready) but
+# EMIT only the town STREET graph. Fidelity cut (charter risk #4, flagged to A who owns the knob):
+# the service/track/footway/path/steps/cycleway tier is EXCLUDED — in Katoomba that's 1461 of 2506
+# ways (Blue-Mountains bushwalking tracks, sidewalks, stairs, driveways, parking aisles), noise for a
+# town street graph and a way-count A's lift would build 2500+ edges from. The full set stays in the
+# raw response if A wants a tier back. `kind` ships as the raw OSM highway class; A maps it (ROAD_KIND).
+ROAD_HW = ("motorway|trunk|primary|secondary|tertiary|unclassified|residential|living_street|road"
+           "|service|track|pedestrian|footway|path|steps|cycleway")
+STREET_KINDS = {"motorway", "trunk", "primary", "secondary", "tertiary", "unclassified",
+                "residential", "living_street", "road", "pedestrian"}
+
 # The big trademarked charity/franchise chains get loving parodies (ported from thriftgod, John's call
 # for the live game); independent shops keep their real OSM names (factual ODbL data).
 PARODY = [
@@ -109,7 +121,49 @@ def fetch_raw(key, bb, refetch):
     return els, fetched
 
 
-def process(key, cfg, els, fetched):
+def fetch_roads_raw(key, bb, refetch):
+    """Return the raw OSM highway ways (with geometry) for the town bbox, cached in-repo like the shops."""
+    os.makedirs(RAW_DIR, exist_ok=True)
+    path = os.path.join(RAW_DIR, key + ".roads.overpass.json")
+    if os.path.exists(path) and not refetch:
+        return json.load(open(path))["elements"]
+    q = (f'[out:json][timeout:120];'
+         f'way["highway"~"^({ROAD_HW})$"]'
+         f'({bb["minLat"]:.5f},{bb["minLon"]:.5f},{bb["maxLat"]:.5f},{bb["maxLon"]:.5f});'
+         f'out geom;')
+    print(f"  [{key}] fetching Overpass roads (highway ways, bounded)")
+    req = urllib.request.Request(OVERPASS, data=urllib.parse.urlencode({"data": q}).encode(),
+                                 headers={"User-Agent": UA})
+    with urllib.request.urlopen(req, timeout=150) as r:
+        ways = json.load(r).get("elements", [])
+    json.dump({"fetchedAt": datetime.date.today().isoformat(), "endpoint": OVERPASS, "query": q,
+               "bbox": bb, "elements": ways}, open(path, "w"), indent=1)
+    time.sleep(2)
+    return ways
+
+
+def roads_from_ways(ways):
+    """OSM highway ways → roads[] (contract v2): {id, kind, name?, pts:[[lat,lon],…]}. Emit only the
+    STREET tier; ship full geometry (A simplifies with Douglas–Peucker); deterministic (id-sorted)."""
+    roads = []
+    for w in ways:
+        hwy = w.get("tags", {}).get("highway")
+        geom = w.get("geometry")
+        if hwy not in STREET_KINDS or not isinstance(geom, list) or len(geom) < 2:
+            continue
+        pts = [[round(g["lat"], 6), round(g["lon"], 6)] for g in geom
+               if isinstance(g.get("lat"), (int, float)) and isinstance(g.get("lon"), (int, float))]
+        if len(pts) < 2:
+            continue
+        rd = {"id": w["id"], "kind": hwy, "pts": pts}
+        name = w.get("tags", {}).get("name")
+        if name:
+            rd["name"] = name[:60]
+        roads.append(rd)
+    return sorted(roads, key=lambda r: r["id"])
+
+
+def process(key, cfg, els, fetched, roads=None):
     center = cfg["center"]
     rows, seen = [], set()
     for el in els:
@@ -144,14 +198,19 @@ def process(key, cfg, els, fetched):
         shops.append({"id": r["id"], "name": pname, "type": TYPE_MAP.get(r["osm"], "opshop"),
                       "lat": round(r["lat"], 6), "lon": round(r["lon"], 6), "suburb": r["suburb"]})
     bb = bbox(center, cfg["span_km"])
-    return {
+    counts = {"raw": len(els), "shops": len(shops)}
-        "schema": "procity-town-cache/1", "key": key, "town": cfg["town"], "source": "osm",
+    cache = {
+        "schema": "procity-town-cache/2" if roads else "procity-town-cache/1",
+        "key": key, "town": cfg["town"], "source": "osm",
         "license": "ODbL 1.0", "attribution": "© OpenStreetMap contributors",
         "generator": "pipeline/build_towns.py", "fetchedAt": fetched,
         "center": {"lat": center[0], "lon": center[1]},
-        "bbox": bb, "counts": {"raw": len(els), "shops": len(shops)},
+        "bbox": bb, "counts": counts, "shops": shops,
-        "shops": shops,
     }
+    if roads:
+        counts["roads"] = len(roads)
+        cache["roads"] = roads                  # schema v2 — A's plan_osm builds the real street graph
+    return cache
 
 
 def span_km(shops):
@@ -166,24 +225,35 @@ def span_km(shops):
 def main():
     args = [a for a in sys.argv[1:] if not a.startswith("-")]
     refetch = "--refetch" in sys.argv
+    no_roads = "--no-roads" in sys.argv          # build a v1 cache (marched fallback), no road fetch
     keys = args or list(TOWNS)
     os.makedirs(TOWNS_DIR, exist_ok=True)
     kept, dropped = [], []
     for key in keys:
         cfg = TOWNS[key]
-        els, fetched = fetch_raw(key, bbox(cfg["center"], cfg["span_km"]), refetch)
+        bb = bbox(cfg["center"], cfg["span_km"])
-        cache = process(key, cfg, els, fetched)
+        els, fetched = fetch_raw(key, bb, refetch)
+        roads = None
+        if not no_roads:
+            roads = roads_from_ways(fetch_roads_raw(key, bb, refetch))
+        cache = process(key, cfg, els, fetched, roads)
         n, sp = len(cache["shops"]), span_km(cache["shops"])
         types = {}
         for s in cache["shops"]:
             types[s["type"]] = types.get(s["type"], 0) + 1
-        status = "OK " if n >= 6 and sp < 5.0 else "LOW"
+        ok = n >= 6 and sp < 5.0
-        if n >= 6 and sp < 5.0:
+        if ok:
             json.dump(cache, open(os.path.join(TOWNS_DIR, key + ".json"), "w"), indent=1)
             kept.append(key)
         else:
             dropped.append((key, n, sp))
-        print(f"  {status} {key:16} raw={cache['counts']['raw']:>4} shops={n:>3} span={sp:.2f}km  {dict(sorted(types.items()))}")
+        rstr = ""
+        if roads is not None:
+            rk = {}
+            for r in roads:
+                rk[r["kind"]] = rk.get(r["kind"], 0) + 1
+            rstr = f"  roads={len(roads)} {dict(sorted(rk.items(), key=lambda x: -x[1]))}"
+        print(f"  {'OK ' if ok else 'LOW'} {key:16} shops={n:>3} span={sp:.2f}km {dict(sorted(types.items()))}{rstr}")
     print(f"\nBUILT {len(kept)} town cache(s) → web/assets/towns/: {', '.join(kept)}")
     if dropped:
         print(f"DROPPED (< 6 shops or span ≥ 5 km): {dropped}")

web/assets/towns/SOURCES.md

@@ -27,10 +27,20 @@ the "© OpenStreetMap contributors (ODbL)" credit visible.
 
 ## Raw responses (`_raw/`)
 
-The raw Overpass JSON for each town is cached in `_raw/<key>.overpass.json` (wrapped with `fetchedAt`,
+The raw Overpass JSON is cached in-repo (wrapped with `fetchedAt`, `endpoint`, `query`, `bbox`):
-`endpoint`, `query`, `bbox`). **Re-runs reuse the raw cache and never re-hit Overpass** — the fetch is
+`_raw/<key>.overpass.json` (shops) and `_raw/<key>.roads.overpass.json` (OSM highway ways, schema v2).
-outward-facing network (John-authorized scout, once per town). `--refetch` forces a deliberate re-fetch.
+**Re-runs reuse the raw cache and never re-hit Overpass** — the fetch is outward-facing network
-`selfcheck.js` reads only top-level `*.json`, so `_raw/` is never loaded as a cache.
+(John-authorized scout, once per town). `--refetch` forces a deliberate re-fetch. `selfcheck.js` reads
+only top-level `*.json`, so `_raw/` is never loaded as a cache.
+
+## Roads — schema v2 (ROUND18, v4.0-alpha REAL ROADS)
+
+A v2 cache carries `roads[]` — the town's real OSM street geometry (`highway=*` ways), same ODbL licence
+and attribution as the shops. **Fidelity cut** (charter risk #4): the raw fetch grabs every `ROAD_KIND`
+tier, but the emitted `roads[]` is the **town street graph only** — drivable roads + pedestrian malls;
+`service`/`track`/`footway`/`path`/`steps`/`cycleway` are excluded (in Katoomba, 1461 of 2506 ways —
+bushwalking tracks, sidewalks, stairs, driveways). The excluded tier stays in the raw response if a
+tier is wanted back. `katoomba_real` is v2 (the alpha); the other towns' roads land as data-only.
 
 ## Names — parody vs factual