diff --git a/docs/00-dependency-graph.md b/docs/00-dependency-graph.md index 2c9c6d7..44bcfea 100644 --- a/docs/00-dependency-graph.md +++ b/docs/00-dependency-graph.md @@ -274,48 +274,103 @@ appear in graph 2 — they need iteration 11 as well as 18. ## 5. porch — the web framework track -States live on [the board's porch section](stories/00-status.md); every -pending node is `readiness: refine` (no approved spec), so a brainstorm -precedes any plan. Three independent roots: **2** (the auth chain), -**6** (the streaming chain), **5** (anytime, no incoming edges). **9** is -the track's only `readiness: ready` item and is held. +States live on [the board's porch section](stories/00-status.md). **The whole +track (2–8) is `readiness: ready`** as of the 2026-09-06 brainstorm; **1** is +done, **9** is held (blocked on the lang-41 arena hang, not an enhancement). +Three independent roots: **2** (the auth chain), **6** (the streaming chain), +**5** (anytime, no incoming edges at all — not even iteration 2). + +This graph makes the **cross-track language edges** visible: the three builtins +the track needs, each drawn as a `lang` node feeding the story that owns it. ```mermaid flowchart TD classDef done fill:#1a7f37,color:#fff,stroke:none - classDef refine fill:#eac54f,color:#000,stroke:none + classDef ready fill:#0969da,color:#fff,stroke:none classDef held fill:#6e7781,color:#fff,stroke:none classDef lang fill:#8250df,color:#fff,stroke:none - CSPRNG["language track: CSPRNG builtin (id 96+) — porch 2's Phase A"]:::lang - P1["porch 1 store-backed middleware ✅ 2026-08-30"]:::done - P2["porch 2 randomness + cookies (repeated Set-Cookie/Vary headers, Cookie: parsing, signed cookies)"]:::refine - P3["porch 3 sessions"]:::refine - P4["porch 4 CSRF"]:::refine - P5["porch 5 routing + response ergonomics"]:::refine - P6["porch 6 streaming core"]:::refine - P7["porch 7 SSE + compression"]:::refine - P8["porch 8 static files + lifecycle"]:::refine - P9["porch 9 idempotent replay (⏸ hold; readiness: ready)"]:::held + RB["language work: random_bytes builtin (bare-name, id 84/90) — porch 2 Phase A"]:::lang + DFL["language work: deflate + crc32 builtins (C) — porch 7 Phase C"]:::lang + TU["language work: time.utc builtin (gmtime sibling of time.local) — porch 8 Phase A"]:::lang - CSPRNG --> P2 + P1["porch 1 store-backed middleware ✅ 2026-08-30"]:::done + P2["porch 2 randomness + cookies"]:::ready + P3["porch 3 sessions"]:::ready + P4["porch 4 CSRF"]:::ready + P5["porch 5 routing + response ergonomics (zero upstream deps)"]:::ready + P6["porch 6 streaming core"]:::ready + P7["porch 7 SSE + compression"]:::ready + P8["porch 8 static files + lifecycle"]:::ready + P9["porch 9 idempotent replay (⏸ hold; readiness: ready)"]:::held + L41["language 41 actor-arena hang (in-progress)"]:::held + + RB --> P2 P2 --> P3 P2 --> P4 P3 --> P4 P6 --> P7 P6 --> P8 - P2 --> P7 P5 --> P7 + P5 --> P8 + DFL --> P7 + TU --> P8 + L41 -.blocks.-> P9 P1 -.re-scope 79e6da4: replay-on-retry split out of 1.-> P9 ``` -The two non-obvious edges are stated in porch 7's own story: gzip's -`Accept-Encoding` negotiation reuses the q-value ranking iteration 5 adds, -and `Vary: Accept-Encoding` needs iteration 2's repeated-header work to -accumulate with other `Vary` contributions. Porch 2's Phase A is -language-track work (the CSPRNG builtin) — the cross-track edge this graph -exists to make visible. Streaming responses' runtime prerequisites -(fibers, iteration 11) are already green in graph 2. +Edges corrected by the 2026-09-06 brainstorm: `P5 --> P7` (gzip's +`Accept-Encoding` reuses iteration 5's q-value ranking) stays, but the old +`P2 --> P7` edge is **gone** — story 7 decided `Vary` accumulates by comma-join +(iteration 5's shape), not iteration 2's repeated-header work. `P5 --> P8` is the +`Download`/`Attachment` helper. The three `lang` nodes are the track's entire +language bill; each is a builtin with a named consumer, none shipped as +decoration. + +## 5a. porch's language-driven gaps (out-of-scope features and the language stories that own them) + +These are the features fiber ships that porch deliberately does **not** — each +excluded because a language primitive does not exist yet. Every edge points from +the owning language story to the porch feature it would unblock (see the +[porch↔fiber scope-gap analysis](plan/exploration/fiber/01-porch-vs-fiber-scope-gap.md)). + +```mermaid +flowchart LR + classDef done fill:#1a7f37,color:#fff,stroke:none + classDef refine fill:#eac54f,color:#000,stroke:none + classDef held fill:#6e7781,color:#fff,stroke:none + classDef gap fill:#cf222e,color:#fff,stroke:none + + L29["language 29 @derive (⏸ hold)"]:::held + L38["language 38 net.connect (refine)"]:::refine + L43["runtime-v2 8 symmetric cipher (refine, NEW 2026-09-06)"]:::refine + L30["runtime-v2 7 observability (refine, moved from language 30, 2026-09-06)"]:::refine + L18["language 18 TTL cache + transaction{} (⏸ hold)"]:::held + L31["language 31 cancellation ✅ (landed in 24)"]:::done + + BIND["typed request binding (fiber Bind)"]:::gap + PROXY["reverse proxy + outbound HTTP client"]:::gap + ENC["encrypted cookies (fiber encryptcookie)"]:::gap + METRICS["metrics / pprof / expvar endpoints"]:::gap + CACHE["cache middleware + recovery rollback"]:::gap + TIMEOUT["per-handler timeout + streaming backpressure"]:::gap + + L29 --> BIND + L38 --> PROXY + L43 --> ENC + L30 --> METRICS + L18 --> CACHE + L31 --> TIMEOUT +``` + +31 (cancellation) is already green — per-handler timeout and streaming +backpressure are unblocked at the language level and wait only on a porch slice +to consume them. The other five gaps are gated on an upstream story: two +brand-new runtime-v2 iterations (8 cipher, 7 observability — moved out of the +language track 2026-09-06), two language iterations on hold (29, 18), one pending +a spec (38). Landed enablers the +track already consumed — 34 (crypto digests), 36 (bit operators), 35 (net +seams) — are green in graphs 1–3 and not repeated here. ## 6. wmux — the multiplexer track (wmux 1) and its gap chain diff --git a/docs/plan/exploration/fiber/01-porch-vs-fiber-scope-gap.md b/docs/plan/exploration/fiber/01-porch-vs-fiber-scope-gap.md new file mode 100644 index 0000000..acabbf1 --- /dev/null +++ b/docs/plan/exploration/fiber/01-porch-vs-fiber-scope-gap.md @@ -0,0 +1,130 @@ +# porch vs fiber — the scope gap, and who prefers which + +> Companion to [00-fiber-parity.md](00-fiber-parity.md). Written 2026-09-06, +> after the porch track (stories 1–8) was brainstormed to `ready`. It reads +> fiber v3 (`.dev/reference/fiber`, commit `3ca9a9d`) against porch's **approved +> scope** — what ships today, plus what stories 1–8 add — and asks what remains +> missing, what fiber does better, and whether a developer would choose porch. +> An exploration doc: no status banner by convention. + +## The 32 middleware, mapped against approved scope + +Nine already had counterparts before the track. The track adds eight more. The +rest are excluded, and the exclusions divide into *principled* (a different +philosophy) and *blocked* (a primitive porch does not have yet). + +- **Shipped before the track (9):** cors, basicauth, keyauth (bearer), helmet + (SecurityHeaders), hostauthorization (HostAllow), etag, static, recover + (trap=500), logger. +- **Added by the track (8):** compress + sse (story 7), csrf (4), session (3), + favicon + healthcheck + rewrite + skip + redirect (5/8), requestid (5), + limiter (story 1, already done). +- **Blocked on the lang-41 runtime hang (1):** idempotency — built, reviewed, + gate-passed, reverted; it is porch story 9, held, not a design gap. +- **Excluded, owner named (rest):** cache/paginate (language 18), expvar/pprof + (runtime-v2 7 observability), proxy (language 38, needs `net.connect`), + encryptcookie (runtime-v2 8 symmetric cipher), adaptor (no net/http + ecosystem), earlydata (TLS at the + proxy), timeout (per-handler, needs cancellation — language 31), envvar + + responsetime (niche helpers, unscoped). + +## What porch lacks even after the whole track lands + +Ranked by how much a real app feels it. + +1. **Typed request binding.** Fiber's `Bind().Body/Query/Cookie(&struct)` is the + single biggest ergonomic gap. porch cannot do it without reflection + (principle 13 forbids it) until `@derive` lands (language iteration 29). Today + a porch handler pulls fields out of maps by hand. This is the one place fiber + is *dramatically* more pleasant, and it is felt on every form and every JSON + endpoint. +2. **Outbound HTTP: client and reverse proxy.** Fiber ships an HTTP client and a + `proxy` middleware. porch has no `net.connect`, so neither exists — an app + that calls another service or fronts one cannot be written in porch at all. + Owner: language iteration 38. +3. **Encrypted cookies.** Fiber's `encryptcookie` is AES-GCM; porch has digests + but no symmetric cipher, so it offers signed-and-readable only. Fine for a + session id, not for a payload an app wants to hide from the client. +4. **Per-handler timeouts.** Fiber wraps a handler in a deadline. porch has + per-*call* net deadlines (iteration 35) but no per-handler timeout, because + cancelling a running handler needs the actor-lifecycle cancellation that + language iteration 31 owns. +5. **Runtime templating.** Fiber renders views at request time against a dozen + template engines. porch rejects this by doctrine — markup is a compile-time + literal (`writeonce-view`). A principled choice, but it rules out the + user-editable-template use case entirely. +6. **The ecosystem.** `adaptor` plugs fiber into all of Go's `net/http` universe; + its session/cache/limiter middleware take pluggable storage drivers (Redis, + Postgres, dozens more). porch has none of that surface and, by single-binary + doctrine, does not want most of it — but it means no drop-in Redis, no + community middleware, no third-party integrations. +7. **Minor, unscoped helpers:** pagination, response-time header, env-var + exposure. Each is a few lines an app can write; none is in the track. + +## What fiber is better at, even where porch has a counterpart + +- **Maturity and ecosystem.** Battle-tested, huge community, pluggable storage + behind every stateful middleware, and the whole Go module world one adaptor + away. This is fiber's decisive, structural advantage and porch will not close + it. +- **Ergonomics.** Binding, generic helpers, reflection-driven convenience — less + hand-written glue per endpoint. +- **Raw performance ceiling.** fasthttp is extreme; a bytecode VM with + single-threaded-per-connection serving will not match its throughput on a + synthetic benchmark. (porch trades this for a different model, below.) +- **Configurability.** Case-sensitivity, strict-slash routing, prefork, storage + backends — many knobs. porch is deliberately opinionated with few. +- **All-in-one networking.** Client, proxy, TLS termination, HTTP/2 in one + process. porch delegates TLS/HTTP2 to a front proxy by doctrine. + +## Where porch is actually better + +The honest counter-case, because "prefer" is not decided on fiber's axes alone. + +- **Durable by default, in one binary.** Sessions, rate-limiting and idempotency + ride the WAL and survive a restart with no Redis, no external store. Fiber's + defaults are in-memory — a restart logs everyone out and resets every counter; + durability means operating a second system. porch's whole stateful surface is + durable with zero extra infrastructure. +- **One static binary, zero dependencies.** Hand-rolled crypto, no CGO, no + driver matrix. Deploy a file. +- **Safety by construction.** No reflection, ownership/borrow checking, and a + house style of secure-by-default (HttpOnly/SameSite defaults, session-id + rotation on login, refusal classes distinguishable in logs but opaque in the + body) and refuse-the-unconstructible (an incoherent heartbeat/idle pair, a + streaming route under header-mutating middleware — both rejected at + construction, never silently half-applied). +- **A concurrency model that fits the hard parts.** Actors/fibers/shards make + SSE fan-out and WebSocket rooms natural rather than bolted on, and idle + connections park for almost nothing. +- **Honesty about limits.** Every gap above is named with an owner; nothing + degrades silently. + +## Would developers prefer porch over fiber? + +It depends on who is asking, and the answer is not "porch wins on features." + +- **A Go developer choosing a framework today: no.** Not on parity terms. Fiber + wins on ecosystem, binding ergonomics, maturity, performance ceiling, storage + flexibility and sheer breadth. porch cannot out-fiber fiber at fiber's own + game, and trying would be the wrong goal. +- **A developer already choosing writeonce: yes, and gladly.** porch is a + coherent, durable-by-default, single-binary framework with strong security + defaults, written *in* the language it serves. Inside the ecosystem there is no + contest — it is the framework, and a good one. +- **A developer choosing on values, either language aside: sometimes.** Someone + who weights one-binary durability with no Redis, memory safety without GC + pauses, opinionated secure defaults, and named-not-hidden limits above + ecosystem breadth may genuinely prefer porch. That is a real but narrow + constituency. + +**The honest positioning.** porch is not a fiber-killer and the approved scope +does not try to be one. It is deliberately *complete enough* to prove the +language can carry a serious web framework, with a distinct thesis — +durable-by-default, single-binary, safe-by-construction — that fiber does not +compete on. Its remaining gaps are almost all one upstream primitive away +(`net.connect` → client/proxy, `@derive` → binding, a cipher → encrypted +cookies, cancellation → handler timeouts), which means the ceiling is set by the +language track, not by porch's design. Developers will prefer porch when they +have already bought the thesis; they will prefer fiber when they are shopping on +breadth. diff --git a/docs/stories/00-status.md b/docs/stories/00-status.md index cd601ed..fe57a3e 100644 --- a/docs/stories/00-status.md +++ b/docs/stories/00-status.md @@ -54,7 +54,13 @@ settled", so a held iteration with an approved spec was indistinguishable from one nobody had thought about. **The startable set is `readiness: ready` and `status: pending`.** -**As of the 2026-08-27 sweep that set has exactly one member:** +**Startable-set counts below are STALE** — the paragraph that follows is the +2026-08-27 sweep and predates the entire **wmux** (23 rungs, ~17 done/partial +as of 2026-09-05) and **runtime-v2** (6 iterations, all done) tracks, plus +lang 42. Treat the per-track tables further down as the current truth; this +snapshot is kept for its explanation of the two-axis model, not its numbers. + +**As of the 2026-08-27 sweep (STALE — see above):** [databasev2 4, io_uring group-commit](databasev2/04-io-uring-commit.md) — its four forks were confirmed settled on 2026-08-20 and nothing has started. Across 47 iterations: 19 done, 5 in-progress, 15 pending, 8 hold; 27 `ready`, 20 @@ -70,6 +76,287 @@ behind this board; live Obsidian Dataview views: ## ▶ NEXT PLAN +### Brainstormed 2026-09-06 — the porch track (2–8) and language 41's fix, both to `ready` + +**What happened this session (docs only, no code):** the whole +[porch track](porch/00-story.md) 2–8 was brainstormed to `readiness: ready` +against `.dev/reference/fiber`; the track's language bill is three small builtins +(`random_bytes`, `deflate`/`crc32`, `time.utc`), plus two gaps moved out to +runtime-v2 ([7 observability](runtime-v2/07-observability.md), +[8 symmetric cipher](runtime-v2/08-symmetric-cipher.md)). And +[language 41](language-runtime-database/41-actor-arena-crash.md) — the arena +hang — was **root-caused and its fix designed to `ready`**. + +**Language 41, settled:** the hang is a **double free from a broken invariant**. +`wo_db_rpc` marshals ("VM heaps are never read cross-shard"), but cross-shard +actor `send`/`call` pointer-shares the message into the receiver's shard — a +worker then drops an object in the sender's arena. **Fix = marshal cross-shard +messages** (copy into the receiver's arena, matching the DB RPC), which +eliminates the class by construction without needing the exact aliasing site; +**plus** aligning the `shard_id % nshards` route/compare mismatch and asserting +`shard_id < nshards`. Poison-on-free and a minimal corpus fixture are named +follow-ups. Proven against `archive/porch-idempotency` 18a–18h/19. + +**Next steps:** implement language 41's marshal fix (unblocks +[porch 9](porch/09-idempotent-replay.md), already written); then porch is +buildable — [porch 5](porch/05-routing-response-ergonomics.md) has zero upstream +deps and is the natural start, with `random_bytes` (porch 2) opening the 3→4 +chain. + +### Landed 2026-09-04 — wmux: switch-client + choose-session (rung 21) + +**Implemented (2026-09-04):** an in-session `switch-client -t B` / `-l` +moves a live client between sessions — the Input `call`s the session (sync, +so a refusal keeps the client on A), a `reg` handle threaded into every +session resolves B, the fds hand off WITHOUT closing, B adopts them and +spawns a fresh Input, the old Input exits only on success; B-occupied +refuses, missing→error, per-session `last_session` for `-l`. Plus a native +**`choose-session` picker** (bound `prefix o`): lists sessions on the status +row, a digit switches. Brainstormed + story-refined first (readiness ready), +then built. Landmine: a `?actor RMsg` nullable field reordered the checkpoint +schema (spurious restart migration) — fixed with a non-nilable `me` + +`DeadReg` placeholder. Gate 52 → 54/0; committed + reinstalled. Full fuzzy +`sesh connect` (fzf + zoxide/config dirs) still needs the tmux-compatible +CLI shim — that's rung 23. + +### Landed 2026-09-04 — wmux: theming, active-pane border, automatic-rename + +**Implemented (2026-09-04):** three developer-requested UI features, each +gated (`just wmux` 50 → 52/0) + committed + reinstalled. +- **Theming (rung 22, first slice)** — a tmux-style style engine + (`style_sgr`: `fg=`/`bg=` named/`bright*`/`colourN`, plus + bold/dim/italic/underscore/reverse), read from durable options + (`opt_style`). Applied to `status-style`, `window-status-current-style`, + `pane-active-border-style` — all `set-option`-able, restart-durable. +- **Active-pane border** — the split divider is now the active border: + drawn in `pane-active-border-style` (green default) as box-drawing, with + a marker pointing at the focused pane (`◄`/`►` / `▲`/`▼`). Focus changes + (click / `select-pane` / cycle) redraw so it follows immediately. This + answers the "panes not clickable" report — clicks always worked (the + cursor moved), there was just no visible cue. +- **Automatic-rename** — the VTE captures the pane's OSC 0/1/2 title; the + Window pushes it to a non-durable `AutoName` table; `window_list` shows + it unless a manual `rename-window` overrides. So starship/vim/bash + setting the title renames the window (refreshed on the status tick). + +**Also fixed same day:** the popup mouse wheel (was dropped — now forwarded +to the popup's app as translated SGR) and mouse re-arm after a full-screen +app disables it (`\e[?1000l` on exit); both gated. + +**`.dev/reference`:** the developer's live lazydocker session + screenshots; +tmux's `pane-active-border-style` / `automatic-rename` / style-string shape. + +**Still the big ones (each its own focused run, NOT rushed):** N-way panes +(3+) + break/swap-pane (rung 10 full), sesh `switch-client` (20), plugin +ports thumbs/fzf/fzf-url (21), control-mode `%notifications` (14), +run-shell/if-shell, pane/layout persistence (16). And the rest of the new +[mouse-UX rung 19](wmux/19-mouse-ux.md): status-line click → window, +drag-resize, drag-select. + +### Landed 2026-09-04 — wmux: lazydocker popup renders clean (OSC + flicker) + +**Reported:** the developer ran `lazydocker` via `display-popup`; it showed +`8;;` garbage smeared across every border/panel and flickered — while the +same lazydocker in tmux was pixel-clean. + +**Root-caused (measured, not guessed):** captured lazydocker's real pty +bytes and replayed a 24 KB slice through the VTE. Two gaps, both gated now: +- **OSC dropped.** lazydocker wraps every bordered element in an OSC-8 + hyperlink (`\e]8;;URI\e\…\e]8;;\e\`). The ESC dispatch had no `\e]` arm, + so it dropped `\e]` and printed the `8;;` payload as cells. Now `\e]…` + consumes the string to its `ST`/`BEL` terminator (covers OSC 0/2 title + + OSC 52 clipboard); `\e(`/`\e)`/`\e*`/`\e+` eat their charset byte too (no + literal `B`/`0`). The garbage was **repaint-only** — a focused pane + passes OSC raw to the client's own terminal, which handles it. +- **Flicker.** `PopReader` read 4 KB, so a ~24 KB frame split into ~6 + partial repaints. It now reads 64 K and drains already-available bytes + (3 ms poll) → one frame, one paint. + +**Verified:** replay of the real capture shows box-drawing/colours/text +intact, zero `8;;`. New gate leg runs an OSC-8 line in a popup (the +repaint path) and asserts clean. `just wmux` **46 → 47/0**. Binary +rebuilt (`woc build … -o wmux`) + reinstalled to `~/.local/lib/wmux/wmux`. + +**`.dev/reference`:** a live `lazydocker` capture via a python pty harness; +the developer's screenshots (wmux vs tmux side by side). + +**Next:** the big remaining parity items are unchanged (N-way panes, sesh +switch-client, plugin ports) — see the marathon entry below. + +### Landed 2026-09-03 — wmux tmux-parity marathon (7 rungs, all gated) + +**Implemented (2026-09-03, "complete all rungs" multi-rung push):** the +switch-blocking tmux features, each gated (`just wmux` 45 → 46/0) + +committed + reinstalled to `~/.local/lib/wmux/wmux`, in dependency order: +- **Rung 10 core** — `split-window -h` (side-by-side) + `-v`; directional + `select-pane -L/R/U/D` (h/j/k/l); zoom (`resize-pane -Z`). Fixed a + spawn-time winsize race (spawn panes at their band size). +- **Rung 12** — system **clipboard** (OSC 52 on yank, verified base64); + **mouse** (SGR enable on attach, wheel→copy-scroll, click→select-pane). +- **Rung 20** (committed at the time as "rung 19") — **display-popup**: a + session-owned modal float running a command (lazygit/lazydocker/sesh) in + a bordered box, reaped on exit. +- **Rung 13** — copy-mode **char selection** (vi `v`/`y`, highlighted, + multi-line range yank → buffer + OSC 52). +- **Rung 15** — terminfo-lite: accept the common TERM families (tmux/ + screen/alacritty/kitty/…), still refuse `dumb`. +- **Rung 17 (full)** — **`#(shell-command)`** (cached, `time.after`- + refreshed) PLUS the recursive expander: `#{?cond,a,b}` conditionals, + `#{b:}`/`#{d:}` modifiers, `#{time}` clock, `#{host_short}`. +- **Window names** — durable `rename-window`, shown as `[idx:name*]`, + `#{window_name}`; **last-window** (`prev`). +- (Earlier same day) **Rung 18** — key tables (`bind-key -n`, Meta/named + keys); and the screen-completeness cluster (UTF-8, sizing, alt-screen, + erase 0/1, SGR reset, O(n log n) replay). + +**Config:** `~/.config/wmux/wmux.conf` maps the developer's tmux binds +(prefix C-a, `-n M-h/M-l`, h/j/k/l select-pane, z zoom, %, G/D/T popups), +auto-loaded by the launcher. + +**Daily-drivable now on xterm.** Remaining for FULL parity (larger, each +its own effort) — rung numbers corrected to final: **N-way panes (3+)** + +break/swap-pane (rung 10 full); ~~format conditionals/modifiers (17)~~ and +~~theming (22)~~ and ~~sesh switch-client (21)~~ have since landed; +**plugin ports** thumbs/fzf/fzf-url + the tmux-compat CLI shim (rung 23, +popup+capture-pane ready); control-mode `%notifications` (14), +run-shell/if-shell, pane/layout persistence (16). + +**`.dev/reference`:** the developer's `~/.tmux.conf` + plugins (the target +config), tmux `popup.c`/`window-copy.c`/`tty.c` shapes, runtime `proc`/ +`term`/`time` seams. + +### Landed 2026-09-03 — wmux screen-completeness + rung 18 (key tables, first slice) + +**Implemented (2026-09-03):** a burst of real-usage fixes driven by +running wmux with the developer's live starship/eza + tmux setup, plus +the first slice of the new [key-tables rung 18](wmux/18-key-tables.md). +- **Dynamic sizing** — the session sizes to the client's terminal via + `term.size` at attach (was a fixed 80×23 box); gate 120×40 → 39×120. +- **Screen completeness (vte.wo)** — the alternate screen (`\e[?1049h/l`, + so btop/vim stop bleeding into the shell), scroll region + cursor + save/restore, erase-display modes 0/1 (`\e[J` clears stale lines below + the cursor), a per-row SGR reset (no colour-bleed blank rows), and + **UTF-8 decoding** (one cell per glyph via `term.width` — nerd-font/CJK/ + emoji render instead of `000`). +- **O(n log n) boot replay** — killed an O(n²) startup CPU burst. +- **Rung 18 slice** — a no-prefix `RootBind` table, `key_code` (Meta + + named keys), a tty key decoder in the Input actor, `bind-key -n`/`-T`. + +**Key findings (measured):** a byte-based VTE mangles every multi-byte +glyph — `feed()` now decodes UTF-8 lead+continuation bytes into one cell +and `term.width(cp)` sets the advance. `\e[K` erases with the CURRENT +SGR, so an un-reset colour painted whole blank rows once the grid filled +the screen. And a **compiler bug** surfaced: `self.f = self.f .. x` +(self-referential field concat-assign) miscompiles — worked around with a +local temp, `emit.ml` fix tracked. Gate `just wmux` 37 → 42, 0 failures. + +**Dependencies unblocked:** rung 18's decoder + tables are the seam the +copy-mode-vi table and `-r` repeat extend; UTF-8 + sizing make the VTE +usable for real prompts/TUIs. The audit's siblings have since landed with +final numbers — 17 formats-v2, 20 display-popup, 21 sesh switch-client, 22 +theming all DONE; 23 plugin ports (the tmux-compat CLI shim) remains. + +**Next steps:** finish rung 18 (copy-mode-vi + `-r`, with rungs 10/13), or +the paused rung 10–22 story map; the `emit.ml` self-concat fix is a +standalone language follow-up. + +**`.dev/reference` used:** the developer's own `~/.tmux.conf` + plugins +(the config the fixes had to render), and the runtime's `term`/`sysio` +seams (`term.size`/`term.width`, EIO). + +### Landed 2026-09-02 — wmux 16 (first slice): the Window owns + reaps its panes + +**Implemented last time (2026-09-02):** the pane-ownership + reaping fix, +a discrete slice of [rung 16](wmux/16-durability-polish.md) surfaced by +live usage — a pane whose child exited on its own (a shell exiting, a +command pane finishing) was left a `` zombie until the session +was killed. The `Window` actor now spawns its OWN panes: `make_window` +and `do_split` `call` the Window (`kind 0` / `kind 6`), it `spawn_pane`s +in-actor (so `owner_actor` = the Window) and returns the reader fd. A +pane's Reader, on EOF/EIO, sends `kind 8`; the Window `wait_dl`s the +child on its own shard to reap it and marks it dead. `DeadWin` and the +`kind 9` replay-feed became dead code and were removed. Gate leg +`attach-zombie` added; `just wmux` 36 → 37, 0 failures. + +**Key findings (measured, not asserted):** the leak was structural, not +a missing `wait()` — the runtime's child slots are PER-SHARD, so +`proc.wait_dl(id)` only works from the actor that spawned the child. +Instrumentation proved it: two live panes reported the SAME shard-local +id (64), and the reader's `wait_dl` trapped `"process id is not a live +child"` because it ran on a different shard. So the kill-time `wait_dl` +was silently failing too; only owner-actor death (kill-session, server +exit) was actually reaping. Moving the spawn into the Window put the +child and its reaper on one shard. Verified: after a command pane and a +window shell both exit, the server has zero defunct children; 0 traps. + +**Learned:** an actor cannot fetch its own address (no self primitive), +so the Reader's window address is threaded from the spawn site — but the +child fd can ride back through a synchronous `call` return (WMsg receive +returns Int), which let the Window own the spawn while the caller (which +holds the window address) wires the Reader. That `call`-returns-a-fd +shape is the clean way to keep ownership and wiring in the right actors. + +**Dependencies unblocked:** the rest of rung 16 (pane/layout +persistence, killw compaction, named buffers) is unchanged and still +pending. The Window-owns-panes shape also makes per-pane resize and +future pane persistence cleaner (the Window is now the single owner). + +**Next steps:** the remaining wmux burn-down — rung 10 (layout tree) or +rung 12 (resize + mouse); then cherry-pick the wmux track dev→master +when the ladder is declared ready. + +**`.dev/reference` used:** the runtime's own `sysio.c` +(`WO_B_PROC_WAIT_DL`, `proc_slot_by_id`, `owner_actor`) — to source the +per-shard child-ownership model that dictated the fix. + +### Landed 2026-09-02 — wmux 11: options/formats/keys + two baseline bug fixes + +**Implemented last time (2026-09-02):** wmux +[rung 11](wmux/11-formats-options-keys.md) — behaviour became durable +DATA. A `Setting {key, val}` options table (`set-option`, `wmux_opt`) +and a `Bind {key, cmd}` key table (`bind-key`), both seeded idempotently +at boot and replayed after a restart; a `#{...}` status-format expander +(`format()`); and ONE `run_command`/`run_session_command` dispatcher +that the CLI, control mode, the `C-b :` prompt, key bindings and the +`WMUX_CONF` config file all feed. Folded rung 14's command-prompt race +fix by moving the line editor into the Input actor. Pure `.wo`, zero +runtime work. `just wmux` 30→36 checks, 0 failures. + +**Key findings (measured, not asserted):** the tmux options/format/keys +DSL (~12k lines in tmux) collapses to two durable tables, a ~40-line +`#{...}` walker, and one dispatcher — and unlike tmux the config SURVIVES +a server restart (proven: `set-option prefix C-t` + `bind-key X` are in +effect after SIGTERM). The prompt race the command-pane rung disclosed is +gone: with the flag and keystrokes in one actor, `C-b : split ` runs +the whole line. The gate's old "command prompt neww" leg was a false +positive — its `[1` needle matched an ANSI cursor escape, not a window; +the new legs assert on the expanded format text instead. + +**Learned (two latent baseline bugs the new paths exposed, both fixed):** +(1) `net.read_dl` returns nil on a timeout but TRAPS on a hard error, and +a dead PTY master returns EIO, so the pane Reader's `catch (e) nil; +continue` pinned a core at 100% the moment ANY command pane's child +exited — a pre-existing runaway confirmed identical on the pre-rung-11 +build. Fix: the catch `return`s (the parser allows a `{ … }` catch arm), +stopping the reader like EOF; same guard added to the Input tty read. +(2) `kill-session` fired `kind 4` at every window at once and each +scanned the shared `Chunk where c.sess` bucket and deleted — colliding +cursors trapped WO-5 "no such row"; chunk cleanup moved into the +serializing session. Only reachable once prompt-`neww` opened real +windows. + +**Dependencies unblocked:** rung 14 narrows to the control-mode command +surface + `%notifications` (its prompt-race scope is done). The unified +dispatcher is the seam rungs 12–16 extend (each new verb is added once). + +**Next steps:** the remaining wmux burn-down — rung 10 (layout tree) or +rung 12 (resize + mouse, runtime already ready via rt2 3/6); then +cherry-pick the wmux track dev→master when the ladder is declared ready. + +**`.dev/reference` used:** tmux (`options.c`, `format.c`, `key-bindings.c` +— the option/format/key shapes wmux compresses); the runtime's own +`sysio.c` `read_dl` (to source the EIO-vs-timeout distinction). + ### Landed 2026-09-02 — runtime-v2 COMPLETE: all five iterations in one run **Implemented last time (2026-09-02):** the whole @@ -629,8 +916,9 @@ asset nobody had built. **Next steps:** the live slice is iteration 24, untouched by this. CI is release-only — no workflow runs the gates per change, which remains the -open half of iteration 30 (observability, CI, fuzz — still no story -file). +open half of iteration 30 (observability — now +[runtime-v2 7](runtime-v2/07-observability.md), moved there 2026-09-06; +CI and fuzz are tooling, split out). **`.dev/reference` used:** none — GitHub Actions' own docs and the runner images' glibc versions were the only sources. @@ -1231,8 +1519,10 @@ check mode, and the `internal/` dep boundary (WO-E108). Driver-only. New 2026-09-01. The I/O plane learned sockets in 8/11/35 and files in 6; this track adds the missing third — **processes, terminals, signals** — -five builtin-sized seams, each `runtime/src/` work with a `types.ml` row -as its whole compiler cost (the iteration 42 precedent). Iteration 42 +first five builtin-sized seams, each `runtime/src/` work with a `types.ml` +row as its whole compiler cost (the iteration 42 precedent); the track then +grew a 6th (terminal measurement) and, 2026-09-06, a 7th and 8th (observability, +moved from language 30; a symmetric cipher) both `refine`. Iteration 42 (bounded subprocess, ✅ on `master` 2026-09-01) opened the arc from the language track before it had a name. **All five `readiness: ready`** — one track-wide brainstorm settled every fork @@ -1251,19 +1541,45 @@ starts. Edges in [dependency graph section 6](../00-dependency-graph.md). | 4 | [termios adoption](runtime-v2/04-termios.md) | ✅ **DONE 2026-09-02** — `term.raw/restore`; restore proven a runtime obligation twice (DIV0 while raw, and the double-raw refusal itself) | | 5 | [fd passing](runtime-v2/05-fd-passing.md) | ✅ **DONE 2026-09-02** — `net.send_fd`/`recv_fd`/`connect_unix`; a tty crossed the socket, was raw'd through the received copy and restored at destroy — the wmux handover in miniature | | 6 | [term.size + term.width](runtime-v2/06-term-size-width.md) | ✅ **DONE 2026-09-02** — TIOCGWINSZ read twin (nil = not a tty) and libc wcwidth under C.UTF-8; the only runtime work the whole wmux parity ladder needs | +| 7 | [observability](runtime-v2/07-observability.md) | ⬜ `refine` — **moved here 2026-09-06** from language iteration 30 (`was_language_iteration: 30`). Runtime metrics/gauges, a `pprof`-equivalent profile, stack-trace-on-trap; consumers named (porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry). Forks: counters-only vs profiling, exposition format, pull vs push, trace-on-trap as a separable first slice. Stretches the track's charter (instrumentation, not processes/terminals/signals) — noted in the story | +| 8 | [symmetric cipher](runtime-v2/08-symmetric-cipher.md) | ⬜ `refine` — **created 2026-09-06** from the [porch↔fiber scope-gap](../plan/exploration/fiber/01-porch-vs-fiber-scope-gap.md); a cipher builtin fits the builtin-seam shape. AEAD for encrypted cookies (porch [2](porch/02-randomness-and-cookies.md) out-of-scope) + data-at-rest; extends [34](language-runtime-database/34-crypto-builtins.md)'s digests, needs porch 2's `random_bytes` for nonces. Load-bearing fork: AES-256-GCM (expected, hard constant-time in software) vs ChaCha20-Poly1305 (easier hand-roll, no-dep doctrine fit) | +| 9 | [in-process TLS](runtime-v2/09-in-process-tls.md) | ⬜ `refine` — **created 2026-09-07** from the gap [jarvis](jarvis/00-story.md) surfaces. TLS **both directions** (outbound client + inbound termination), **retiring the "TLS is the proxy's job" doctrine** (recorded in 34/38/porch). The track's heaviest seam — **not** builtin-sized, and likely a **vendored-lib exception** to no-external-deps (TLS is the one thing not to hand-roll). Load-bearing fork (left open): vendor mbedTLS/BearSSL vs hand-roll a subset. Sits on language 38's `net.connect`; gates jarvis entirely | ### ▸ wmux — the terminal multiplexer track -New 2026-09-01, from [the tmux parity study](../plan/exploration/tmux/00-tmux-parity.md). -First of the *softwares built with writeonce* tracks: the product is an -end-user program, not a library. Its runtime prerequisites are the -[runtime-v2 track](runtime-v2/00-story.md) above — iteration 42 was the -first domino; runtime-v2 1–5 remain, streaming-subprocess first — plus -the VTE grid + unicode width work wmux 1 itself owns. +New 2026-09-01, from [the tmux parity study](../plan/exploration/tmux/00-tmux-parity.md); +**re-scoped 2026-09-02 to FULL tmux parity** as a nine-rung ladder +([ladder spec](../superpowers/specs/2026-09-02-wmux-ladder-design.md)), +serving the recorded goal: acceptance by Linux-based developers. Runtime +prerequisites ALL landed (runtime-v2 1–6); every rung past 1 is pure +`.wo`. Durability is the ladder-wide differentiator — every rung's +state replays after a server restart, which tmux loses by design. | # | Iteration | State | | --- | --- | --- | -| 1 | [wmux](wmux/01-wmux.md) *(was language 43)* | ⬜ `refine` — five forks recorded (terminfo, v1 surface without split panes, scrollback residency, command surface, streaming verb shape). The beyond-tmux leg: durable sessions replay layout + scrollback after a server RESTART | +| 1 | [foundation](wmux/01-wmux.md) *(was language 43)* | ✅ **DONE 2026-09-02** — sessions, attach by fd-handover, durable capped chunk log, restart replay; `just wmux` 19/0 under a real PTY harness incl. the beyond-tmux restart-replay leg | +| 2 | [the screen](wmux/02-the-screen.md) | ✅ **DONE 2026-09-02** — VTE grid (vte.wo); reattach/restart paint the screen, proven grid-specific in the gate | +| 3 | [windows + status](wmux/03-windows-and-status.md) | ✅ **DONE 2026-09-02** — actor-per-window, status line, C-b c/n/p/digit; windows durable across restart | +| 4 | [split panes](wmux/04-split-panes.md) | ✅ **DONE 2026-09-02** — vertical 2-pane split, focus, composite grid render (N-way/horizontal deferred) | +| 5 | [copy mode](wmux/05-copy-mode.md) | ✅ **DONE 2026-09-02** — scrollback, copy-mode paging, yank to a durable paste buffer, paste | +| 6 | [multi-client](wmux/06-multi-client.md) | ✅ **DONE 2026-09-02** — multi-client mirroring (min-size/live-resize/mouse deferred) | +| 7 | [command system](wmux/07-command-system.md) | ✅ **DONE 2026-09-02** — C-b : prompt (neww/split/next/prev/killw) + config file (session directive) | +| 8 | [hooks + control](wmux/08-hooks-and-control.md) | ✅ **DONE 2026-09-02** — control-mode line protocol + session-created hooks delivered as actor messages | +| 9 | [parity audit](wmux/09-parity-audit.md) | ✅ **DONE 2026-09-02** — the tmux-vs-wmux catalog: every gap a named follow-up or a refusal by name | +| 10 | [layout tree](wmux/10-layout-tree.md) | 🟡 `refine` — **first slice DONE 2026-09-03**: horizontal `split-window -h`, directional `select-pane -L/R/U/D` (h/j/k/l), zoom (`resize-pane -Z`); rung-22 added the active-pane border marker. Still 2-pane max — N-way (3+), swap/break-pane, presets, durable layout pending | +| 11 | [formats + options + keys](wmux/11-formats-options-keys.md) | ✅ **DONE 2026-09-02** — durable options + `bind-key` tables, `#{...}` status format, one `run_command` dispatcher behind CLI/control/prompt/keys/config; folded rung 14's prompt-race fix. Fixed two baseline bugs: a PTY-EIO reader 100%-CPU spin and a concurrent kill-session chunk race. `just wmux` 36/0 | +| 12 | [resize + mouse](wmux/12-resize-and-mouse.md) | 🟡 `refine` — **attach-time sizing + SGR mouse DONE 2026-09-03/04**: sizes to the client's terminal via `term.size` (was fixed 80×23); wheel→copy-scroll, click→select-pane, status-row click→window; mouse re-armed after a full-screen app. Live SIGWINCH resize + multi-client min-size still pending | +| 13 | [copy selection + search](wmux/13-copy-selection-search.md) | 🟡 `refine` — **char-range selection DONE 2026-09-03**: vi `v`/`y`, highlighted, multi-line yank → buffer + OSC 52. Only incremental search + rectangle select pending | +| 14 | [control surface + prompt fix](wmux/14-control-and-prompt.md) | ⬜ `refine` — broaden control mode. **Prompt-race fix DONE in rung 11**; this rung narrows to the control-mode command surface + `%notifications` | +| 15 | [terminfo](wmux/15-terminfo.md) | 🟡 `refine` — **terminfo-lite DONE 2026-09-03**: a TERM allowlist (xterm/screen/tmux/alacritty/kitty/…; refuses only `dumb`) retired the foreign-`TERM` refusal. Full compiled-terminfo parsing still pending | +| 16 | [durability polish](wmux/16-durability-polish.md) | 🟡 `refine` — pane/layout persistence, killw compaction, buffers STILL pending. **First slice DONE 2026-09-02**: the Window owns + reaps its panes (spawns them in-actor so `wait_dl` works on its shard), fixing a `` zombie leak; gate leg `attach-zombie`, 37/0 | +| 17 | [formats v2](wmux/17-formats.md) | ✅ **DONE 2026-09-03** — the full status format engine: `#(shell)` (cached, timer-refreshed), recursive `#{...}` with `#{?cond,a,b}` conditionals, `#{b:}`/`#{d:}` modifiers, `#{time}`/`#{host_short}`/real `#{window_name}` | +| 18 | [key tables](wmux/18-key-tables.md) *(new, from the config audit)* | 🟡 `refine` — **first slice DONE 2026-09-03**: no-prefix root table (`bind-key -n`), Meta + named keys (`key_code`), a tty key decoder in the Input actor; gate `bind-key -n M-h` fires without prefix, 42/0. `copy-mode-vi` table, `-r` repeat still pending | +| 19 | [mouse-driven UX](wmux/19-mouse-ux.md) *(new)* | 🟡 `refine` — **active-pane border highlight DONE 2026-09-04** (the split divider is the focus border + a direction marker; click flips it) plus **status-row click → select window** (2026-09-04). Still pending: drag-resize, drag-select. Forks open — brainstorm the rest before build | +| 20 | [display-popup](wmux/20-display-popup.md) | ✅ **DONE 2026-09-03/04** — session-owned modal float (`-E`), `-B` borderless (flush app frame), rounded gray border, popup mouse-wheel forwarding; drove the VTE OSC-swallow + popup frame-coalesce fixes. (Committed as "rung 19" then renumbered) | +| 21 | [sesh + switch-client](wmux/21-sesh-switch-client.md) | ✅ **DONE 2026-09-04** — in-session `switch-client -t B` / `-l`: sync `call` hand-off (Input exits only on success), `reg` threaded into sessions, fds handed off without close, B adopts + spawns a fresh Input; B-occupied refuses, missing→error. Plus a native **`choose-session` picker** (prefix `o`) — lists sessions on the status row, a digit switches. Fixed a `?actor RMsg` nullable-wrapper schema reorder (spurious restart migration) with a non-nilable `me`. Gate 54/0. (Full fuzzy `sesh connect` needs the tmux-compat CLI — rung 23) | +| 22 | [theming](wmux/22-theming.md) | ✅ **DONE 2026-09-04** — a tmux-style `style_sgr` engine (fg/bg named/bright/colourN + attrs) read from durable options; applied to `status-style`, `window-status-current-style`, `pane-active-border-style` (active border). Plus **automatic-rename** (window name follows the pane's OSC title). Later surface (`mode-style`, per-window format styling, `message-style`) can extend it | +| 23 | [plugin ports](wmux/23-plugin-ports.md) | ⬜ `refine` — run the developer's tmux plugins (thumbs, fzf, fzf-url) under wmux via `capture-pane` + a popup picker. Forks open (port each vs a `tmux` compat shim, which plugins first, capture scope). Rides rung 20 + rung 12 | ### Language track — sequenced, on the critical path diff --git a/docs/stories/jarvis/00-story.md b/docs/stories/jarvis/00-story.md new file mode 100644 index 0000000..0f69819 --- /dev/null +++ b/docs/stories/jarvis/00-story.md @@ -0,0 +1,107 @@ +# Story — `jarvis`, the writeonce AI assistant + +The sixth track, and the second whose product is an end-user *program* rather +than a language capability — after [`wmux`](../wmux/00-story.md), the terminal +multiplexer. Where wmux proves writeonce can build the tool a developer lives +in, jarvis proves it can build the tool of the moment: an AI assistant, written +end to end in `.wo`, durable and single-binary by construction. It serves the +same north star — adoption by Linux developers — by meeting them where the +attention is. + +Numbering restarts at 1 and is local to this track; frontmatter carries +`track: jarvis`. Status rules are the repo's, unchanged: `status:` in +frontmatter is the only place state lives, no directory encodes it. + +## The problem, stated once + +An assistant's whole job is to reach a model, and the runtime cannot reach +anything outbound. It learned to *listen* — sockets in 8/11/35, unix sockets and +peer address in 35 and runtime-v2 — but it has never learned to *dial*: there is +no `net.connect` (outbound TCP), confirmed against `runtime/src/wob.h` (the net +builtins stop at listen/accept/read/write plus the unix-socket client), and no +outbound TLS client anywhere. An LLM API is HTTPS on a remote host. jarvis +therefore does not begin until two things exist. + +The design chosen is **direct outbound HTTPS** — jarvis dials the LLM API +itself, keeping the pure single-binary story. That gates the whole track on +language work: + +- **`net.connect`** — outbound TCP — + [language 38](../language-runtime-database/38-content-platform-capabilities.md), + written but not yet built. +- **An outbound TLS client** — HTTPS over that socket — now owned by + runtime-v2 [9](../runtime-v2/09-in-process-tls.md) (in-process TLS), created + 2026-09-07 from this gap. It **retires the standing "TLS is the proxy's job" + doctrine**, giving the runtime TLS both directions — the load-bearing choice + jarvis's direct-HTTPS design forced into the open. + +A **local-gateway alternative was considered and set aside**: jarvis could speak +to a small companion process over a unix socket (`net.connect_unix`, id 107) or +spawn one (`proc.spawn`, id 97) — both exist today — and let that companion do +the HTTPS, exactly as inbound TLS terminates at a proxy. It is buildable now. +It was rejected in favour of the single-binary story, in which the assistant +owns its own connection rather than shipping a second executable. + +## Architecture + + browser ⇄ jarvis (a porch app) ⇄ [blocked seam: net.connect + TLS] ⇄ LLM API + +Requests arrive at a porch web app; the answer streams the other way, token by +token, LLM → jarvis → browser, over porch's SSE. Conversation state is durable +in a `@table`, so history survives a restart with no external store — the +writeonce differentiator wmux already showed for session state, applied to chat. + +## The iterations + +Ordered by dependency; the first rung is the whole end-to-end seam, and nothing +past it is worth building until that seam is proven. + +| # | Iteration | Delivers | Needs | +| --- | --- | --- | --- | +| 1 | the chat loop | a prompt sent to one LLM, tokens streamed back to the browser, the conversation persisted durably | the outbound seam (language 38 + TLS); porch 2/3/6/7; wo-html | +| 2 | tool use / the agent loop | function-calling and multi-step orchestration through actors — where "assistant" becomes "agent" | 1 | +| 3 | retrieval (RAG) | embeddings + vector search over a document set; carries its own sub-gap — an embeddings call over the same outbound path, plus a vector store (pure-`.wo` or a new primitive, decided in that story) | 1, and the embeddings/vector decision | + +Sketched, not committed — named so the shape is visible, not to schedule them: +**model routing / multi-model** (choose a backend per request) and an **MCP +client** (jarvis as an MCP host, calling tools over the protocol) — both +on-brand, both later. + +Only iteration 1's scope is settled by this overview; every iteration file is +written and refined to `ready` before its code lands, per the repo's story +discipline. + +## Dependencies + +Consumed, and already `ready` or shipped: + +| Needs | From | +| --- | --- | +| signed cookies, session id | porch [2](../porch/02-randomness-and-cookies.md) | +| durable conversation history, revocable sessions | porch [3](../porch/03-sessions.md) + `@table` | +| incremental response writes | porch [6](../porch/06-streaming-core.md) | +| token streaming to the browser | porch [7](../porch/07-sse-and-compression.md) (SSE) | +| the chat UI | `wo-html` / `writeonce-view` | + +Blockers, which must land before iteration 1 starts: + +| Blocker | Owner | +| --- | --- | +| outbound TCP (`net.connect`) | language [38](../language-runtime-database/38-content-platform-capabilities.md) | +| outbound TLS client | runtime-v2 [9](../runtime-v2/09-in-process-tls.md) — in-process TLS, created 2026-09-07 from this gap; **retires the proxy-termination doctrine** | + +## What this track does NOT own + +| Not jarvis's | Why | +| --- | --- | +| local, in-process model inference | needs an ML runtime and heavy FFI — against the no-external-dependency doctrine | +| the local-gateway companion process | considered and rejected (above) in favour of the single-binary story | +| voice / audio in or out | a separate surface with its own capture and codec story; no rung asks for it | +| starting before the blockers land | like [porch 9](../porch/09-idempotent-replay.md) waiting on language 41, jarvis waits on the outbound seam — documented, not worked around | + +## Review protocol + +Same as every track: the developer reads one iteration, approves or amends, and +the next starts only after approval. Each iteration is an unsplittable value +slice with phases, Given/When/Then acceptance criteria, and an out-of-scope +list, proven by a gate before it is called done. diff --git a/docs/stories/language-runtime-database/41-actor-arena-crash.md b/docs/stories/language-runtime-database/41-actor-arena-crash.md index 84c95c7..1ab7024 100644 --- a/docs/stories/language-runtime-database/41-actor-arena-crash.md +++ b/docs/stories/language-runtime-database/41-actor-arena-crash.md @@ -2,7 +2,7 @@ track: language-runtime-database iteration: "41" status: in-progress -readiness: refine +readiness: ready --- # 41 — the actor arena crash: a SIGSEGV under concurrent parked callers @@ -65,13 +65,229 @@ So the hang is its own defect and needs its own investigation. It was not caught in this pass: a `gdb` attach needs the hung process held open, and eight scripted attempts to catch one in the act did not land inside the time budget. -**Where to look first.** `wo_engine_stop` sets `eng_shutdown`, wakes each +### Status 2026-09-05 — the hang is ROOT-CAUSED. It is a double free. + +Neither of the two candidates below was right, and the third guess in this +file — "two shards route the same payload to each other" — is half right: it is +**one** shard routing to *itself*. + +`gdb` is still unusable here (`ptrace_scope=1` blocks a sibling tracer), so the +evidence came from `/proc` plus counters compiled into the runtime. Two CPU +samples a second apart during a live hang: `Threads: 1`, state `R`, `wchan 0`, +utime 157 → 212 and stime 148 → 193. One thread spinning at 100%, every worker +already joined — so the hang is past `pthread_join`, and `main()` had returned. + +**The livelock, measured.** A counter around `while (eng_settle_inboxes() > 0)` +shows it returning **12, forever**: `moved_total` is exactly 12 × passes at one +million passes. Dumping those envelopes names the mechanism: + +``` +L41 env: inbox=19 vm_rt_shard=19 kind=2 obj_shard=32019 class_id=-3888 nshards=20 +``` + +`obj_shard=32019` is not a shard. It is the **high 16 bits of a pointer** +(`0x7d13`), and `class_id` is that pointer's low 32 bits. `wo_arena_free` frees +a block by writing the freelist next-pointer over its first 8 bytes — which is +exactly `class_id` (0..3), `shard_id` (4..5), `flags`, `pad`. So the payload +being settled is **an already-freed block**, and the header being read is a +freelist link. + +That garbage id is what makes it spin rather than crash or leak. +`wo_route_free` pushes to `INBOX[shard_id % WO_ENG_MAX_SHARDS]`, but +`wo_drop_obj` compares the **unmasked** `shard_id` against `rt->shard_id`. +`32019 % 64 == 19`, so the envelope lands back in the very inbox it came from, +is judged foreign again, and routes again. Any id ≥ 64 congruent to a live +shard mod 64 livelocks the settle loop. + +**Where the freed block enters.** A `backtrace()` on any route whose +`shard_id >= nshards` puts the origin in the RUN, not in teardown: + +``` +main → wo_vm_call → vm_run (vm.c:2163) → wo_vm_adopt (vm.c:174, case 2, + a home-routed free) → wo_drop_obj → class_free (gc.c:66) + → wo_drop_kind → wo_drop_obj → wo_route_free +``` + +The parent object is intact; its **fields** are not: + +``` +L41 BADFIELD: parent class_id=0 shard=0 | field #0 kind=4 target_shard=29774 +L41 BADFIELD: parent class_id=0 shard=0 | field #2 kind=4 target_shard=29774 +L41 BADFIELD: parent class_id=0 shard=0 | field #3 kind=4 target_shard=29774 +``` + +Kind 4 is `WO_K_MULTI`. A shard-0 object owns three `multi` containers that +live in **another shard's arena** and have already been freed. Freeing the +parent drops them a second time. This is why the allocation-heavy arm was the +only one that ever failed and the four-scalar arm never did: **containers +crossing the mailbox are the trigger**, not `call` and not allocation volume as +such. Slot recycling under insert+delete churn only decides how fast the freed +block gets reused, which is why N=4/N=5 looked causal. + +### Which side drops first — settled 2026-09-06 + +Measured, not inferred. The instrument is a per-arena history table: every +`wo_arena_alloc` (both the bump and freelist paths) and every `wo_arena_free` +records the class the block carried plus two return addresses, keyed by block +address. A dangling pointer's history then names the free that orphaned it. The +earlier suspicion — the parked-`call` re-execution dropping its moved argument +twice — was **wrong**. + +The block that starts the livelock has this history, all of it in **shard 1's** +arena: + +``` + -- PARENT 0x73cdc7fff1e0 -- + #0 ALLOC bump via wo_obj_new+0x53 + #1 FREE as user(14) via class_free <- wo_drop_obj+0xcb +``` + +**First drop: the object's own home shard, legitimately.** Shard 1 allocated it +as class 14 and freed it through the ordinary owned-graph path. Nothing is wrong +up to here. + +**Second drop: a worker executing the compiled `DROP` opcode**, walking a +*different* owner graph that still reaches the same block: + +``` +wo_vm_serve → vm_run (vm.c:2153, CASE(DROP)) → wo_drop_obj + → class_free (gc.c:67) → wo_drop_kind → wo_drop_obj + → multi_free (gc.c:42) → wo_drop_kind → wo_drop_obj + → class_free → wo_drop_kind → wo_drop_obj → wo_route_free ← stale +``` + +So the defect is **an owned subtree reachable from two owner graphs**: one shard +1 has already destroyed, one still live on a worker. It is shared where it +should have been transferred. The `multi_free` frame in the middle is why only +the container-carrying arm ever failed. + +**Why the second drop lands on shard 0 rather than trapping.** `wo_arena_free` +writes the freelist next-pointer over the block's first 8 bytes. When the block +is the **tail** of its size class that pointer is NULL, so the header reads back +`class_id 0, shard_id 0, flags 0` — and class 0 is a *valid* class index. The +worker sees `shard_id 0 != its own`, routes the free to shard 0; shard 0 adopts +it (`wo_vm_adopt` case 2), matches `0 == 0`, concludes "we are home", and runs +`class_free` with **class 0's** field kinds over a dead class-14 object. Class +0's kinds say fields #0/#2/#3 are `WO_K_MULTI`; the slots actually hold the dead +object's Text pointers, already freed by shard 1. Those carry freelist links as +headers, so their `shard_id` is a pointer's high 16 bits — 29645, 32019 — and +the modulo alias sends them back to the inbox they came from, forever. + +Three distinct defects, in fix order: + +1. **The aliased subtree** — the root cause. A child owned by two graphs. + Not yet localised to the code path that creates the alias; that is the next + question, and the `DROP` site plus the `multi_free` frame are where to look. +2. **A freed block is indistinguishable from a live class-0 object.** A NULL + freelist link forges a valid header. A poison class id (or a free bit in + `flags`) would turn every one of these into an immediate, named trap instead + of a silent misinterpretation — cheap, and it would have caught this on the + first run. +3. **The modulo alias.** `wo_route_free` pushes to `INBOX[shard_id % 64]` while + `wo_drop_obj` compares the unmasked id. Mask consistently, or refuse to route + an id ≥ `nshards`. + +**Two defects, and the order matters.** Bounding the settle loop would stop the +hang and leave a double free behind, turning a visible spin into a silent +corruption. Fix the ownership bug first; the modulo alias is a real second +defect worth its own fix (mask consistently, or refuse to route an id ≥ +`nshards`), but it is not the root cause. + +Reproduction, all of it scripted: worktree at `archive/porch-idempotency`, +cherry-pick `9dca0b4` onto it, then loop `scripts/web-app-accept.sh`. The +`/proc` dump, the settle counter, the envelope trace and the bad-field trace are +each a few lines against `vm.c` and `gc.c`. + +**Where to look first (superseded — kept for the record).** +`wo_engine_stop` sets `eng_shutdown`, wakes each worker's eventfd, then `pthread_join`s. Two candidates worth eliminating before anything else: a worker blocked in `wo_io_wait` with a fiber parked on a `call` whose reply will never arrive, and the primary spinning in `while (eng_settle_inboxes() > 0) {}` if two shards can route the same payload to each other indefinitely. The second is cheap to rule out with a counter. +## Fix design — settled 2026-09-06 (`readiness: ready`) + +The root cause is a **broken invariant**, not a stray double free. `wo_db_rpc` +states the invariant plainly (`vm.c:281`): *the args are ENCODED into engine +slots on THIS thread — VM heaps are never read cross-shard.* The DB path +marshals. But cross-shard actor `send`/`call` (`vm.c:1098-1112`) does +`e->payload = msg_val` — it **pointer-shares** the message into the receiver's +shard. A worker then reads and eventually drops an object that lives in the +sender's arena, and the double free, the class-0 forge and the modulo livelock +are all downstream of that single violation. + +### Decision 1 — marshal cross-shard messages (root fix) + +Cross-shard `send` and `call` copy the message into the receiver's arena on the +crossing, exactly as `wo_db_rpc` already marshals its args. No pointer crosses +an arena boundary, so the double-free class is eliminated **by construction** — +and, importantly, the exact aliasing *site* need not be localised, because the +fix removes the shared pointer rather than the specific graph that aliased it. +It restores the "heaps are never read cross-shard" invariant the actor path +currently breaks, and it closes the latent hazard beyond the double free: a +worker reading sender-arena fields is unsafe under GC or compaction even when +the ownership happens to be clean. The cost is a copy per cross-shard message — +the same cost the DB RPC already pays, and correctness outranks the zero-copy +the current path was reaching for. Same-shard send is unchanged (the arena is +shared, the pointer move is correct — which is why `WO_SHARDS=1` never failed). + +### Decision 2 — fix the modulo alias and bound the shard id (this iteration) + +`wo_route_free` pushes to `INBOX[shard_id % nshards]` while `wo_drop_obj` +compares the **unmasked** `shard_id` against `rt->shard_id`; that mismatch is +what makes a stale free self-route forever instead of resolving. The two sites +are made consistent, **and** both assert `shard_id < nshards` — an out-of-range +id is impossible for a live object, so hitting it is a corrupt or freed header +and must trap loudly rather than route somewhere. This lands with the root fix +because it is the guard that would have turned the original silent livelock into +an immediate diagnostic. + +### Decision 3 — poison-on-free is a follow-up, not this iteration + +The deeper defensive fix — stamping a freed block's header (a free bit in +`flags`, or a poison `class_id`) so a NULL freelist link can never forge a valid +class-0 object — is deferred to its own story. Decision 2's bounds assert already +catches the specific corrupt-header shape this bug produces at route time; the +general poison is broader and separable. + +### Decision 4 — prove against the existing repro; a minimal corpus fixture is a follow-up + +The marshal fix is proven against the `archive/porch-idempotency` reproduction: +recover the branch, cherry-pick `9dca0b4`, and run `scripts/web-app-accept.sh` +sections 18a–18h and 19 to stability (the idempotency legs that failed one run +in six). A minimal, deterministic corpus fixture — a cross-shard `send` of an +object carrying an owned subtree (`multi`/`Text`), both sides then dropping, +under `WO_SHARDS>1` and ASan — is worth pinning but is its own follow-up; it is +not required to land the fix. + +### Phases + +- **A — marshal.** Make cross-shard `send`/`call` (kinds 0 and 5) copy the + payload into the receiver's arena on the crossing, mirroring `wo_db_rpc`. The + monitor path (kind 7) carries a payload too and is audited the same way. + Same-shard paths untouched. +- **B — the modulo/bounds guard.** Align the `shard_id` comparison in + `wo_drop_obj` with the masking in `wo_route_free`, and assert `shard_id < + nshards` at both the route and the home-check. +- **C — prove and close.** Re-run the archive repro's 18a–18h/19 to stability + under ASan, confirm the settle loop no longer spins, and unblock + [porch 9](../porch/09-idempotent-replay.md). + +### Acceptance criteria + +- **Given** a cross-shard `send`/`call` of an object with an owned subtree, + **when** both the sender's graph and the receiver drop, **then** each block is + freed exactly once and no free is routed across an arena boundary. +- **Given** the `archive/porch-idempotency` repro under `WO_SHARDS>1` and ASan, + **when** sections 18a–18h and 19 run repeatedly, **then** they are stable — + the one-run-in-six idempotency hang is gone — and the settle loop terminates. +- **Given** a header carrying a `shard_id >= nshards`, **when** it reaches the + drop/route path, **then** it traps loudly rather than self-routing. +- **Given** every same-shard workload, **when** the runtime battery and corpus + run, **then** they are byte-for-byte unchanged — the marshal cost falls only + on the cross-shard path. + ## The original symptom Two shapes, believed at the time to share one cause: @@ -136,5 +352,17 @@ fixed. ## Out of scope -Fixing the porch feature that found it. That is [porch 9](../porch/09-idempotent-replay.md), -and it is already written; it only needs this to land first. +- **Fixing the porch feature that found it.** That is + [porch 9](../porch/09-idempotent-replay.md), already written; it only needs + this to land first. +- **Poison-on-free** (decision 3) — a separate defensive story: stamp a freed + header so a NULL freelist link can never forge a valid class-0 object, + trapping any stale drop rather than misreading it. Needs a language-track + number when picked up. +- **A minimal deterministic corpus fixture** (decision 4) — a cross-shard `send` + of an object with an owned subtree, both sides dropping, under `WO_SHARDS>1` + + ASan. Worth pinning; its own follow-up. +- **The two smaller runtime defects found alongside** (above): the + `try EXPR catch (e) nil` Int-0-vs-trap ambiguity and the `json.decode ... as T` + cross-return-boundary corruption. Both worked around in the archived code; + each deserves its own minimal fixture and fix, neither blocks this. diff --git a/docs/stories/porch/00-story.md b/docs/stories/porch/00-story.md index 9dee1f8..4daa21d 100644 --- a/docs/stories/porch/00-story.md +++ b/docs/stories/porch/00-story.md @@ -73,7 +73,7 @@ risky work starts. | typed binding of query/params/form into a class | language: [`@derive`](../language-runtime-database/29-compile-time-metaprogramming.md) — reflection is forbidden by principle 13 | | TTL cache, `transaction { }`, durable job queue | language: [iteration 18](../language-runtime-database/18-memory-db-features.md) | | a `proxy` middleware | language: [iteration 38](../language-runtime-database/38-content-platform-capabilities.md) — needs `net.connect`, which does not exist | -| metrics, profiling, per-change CI, fuzzing | language iteration 30 (no story file yet) | +| metrics, profiling, per-change CI, fuzzing | [runtime-v2 7](../runtime-v2/07-observability.md) — observability (was language iteration 30; metrics/profiling/trace-on-trap; CI + fuzz are tooling, split out) | | TLS, HTTP/2 | nobody — proxy-terminated by doctrine | | a runtime template engine | nobody — rejected; markup is a compile-time literal (`writeonce-view`) | | a radix-tree router | nobody yet — waiting on a *measurement*, not a decision | diff --git a/docs/stories/runtime-v2/00-story.md b/docs/stories/runtime-v2/00-story.md index b76abae..e9258b3 100644 --- a/docs/stories/runtime-v2/00-story.md +++ b/docs/stories/runtime-v2/00-story.md @@ -34,7 +34,17 @@ cross a unix socket. Five seams, each builtin-sized, each in | 4 | [termios adoption](04-termios.md) | the process's OWN tty into raw mode and back — adopting a terminal it was given | | 5 | [fd passing](05-fd-passing.md) | SCM_RIGHTS over unix sockets — detach/attach's foundation, and the Wayland stage's later | -**ALL FIVE LANDED 2026-09-02, one execution run** (plan: +The track then grew past its original five seams — same shape (a builtin in +`runtime/src/` with a `types.ml` row), broader than processes/terminals/signals: + +| # | Iteration | What it adds | +| --- | --- | --- | +| 6 | [term.size + term.width](06-term-size-width.md) | the two terminal-measurement verbs the wmux ladder asked for | +| 7 | [observability](07-observability.md) | metrics/gauges, a `pprof`-equivalent profile, stack-trace-on-trap (moved from language iteration 30, 2026-09-06) — `refine` | +| 8 | [symmetric cipher](08-symmetric-cipher.md) | AEAD (encrypt/decrypt) for encrypted cookies and data at rest, extending iteration 34's digests (moved from language iteration 43, 2026-09-06) — `refine` | +| 9 | [in-process TLS](09-in-process-tls.md) | TLS **both directions** — an outbound client (dial HTTPS) and inbound termination — **retiring the "TLS is the proxy's job" doctrine**; created 2026-09-07 from the gap [jarvis](../jarvis/00-story.md) surfaces. The track's heaviest seam (not builtin-sized; likely a vendored-lib exception) — `refine` | + +**ALL FIVE [1–5] LANDED 2026-09-02, one execution run** (plan: [`2026-09-01-runtime-v2.md`](../../superpowers/plans/2026-09-01-runtime-v2.md); three implementation amendments in the spec's History). Gates: `test_proc` 193/0 + `test_term` 60/0 inside a fully green ASan suite on diff --git a/docs/stories/runtime-v2/07-observability.md b/docs/stories/runtime-v2/07-observability.md new file mode 100644 index 0000000..45133d5 --- /dev/null +++ b/docs/stories/runtime-v2/07-observability.md @@ -0,0 +1,88 @@ +--- +track: runtime-v2 +iteration: "7" +was_language_iteration: "30" +status: pending +readiness: refine +--- + +# runtime-v2 7 — observability: metrics, profiling, and traces on trap + +> Moved 2026-09-06 from the language track (was language iteration 30, the +> number a dozen docs still point at) into runtime-v2, whose builtin-sized-seam +> shape it fits. It stretches the track's original processes/terminals/signals +> charter — observability is runtime instrumentation of the VM, GC and shards — +> but the track already grew past its first five seams. **`readiness: refine`** — +> the gap, its consumers and its forks are named here, nothing is brainstormed to +> `ready` yet. + +## Why this exists + +The runtime has no observability surface. A healthcheck answers one bit +(up/not-up); nothing exposes counters, gauges, latencies, memory, or a +profile. Every attempt to reason about the system's behaviour at runtime hits +the same wall, which is why the number is referenced from five directions at +once: + +- **porch** excludes `expvar`, `pprof` and metrics endpoints by pointing here + ([story 8](../porch/08-static-and-lifecycle.md), + [39](../language-runtime-database/39-web-framework-parity.md)) — a healthcheck is one bit, not + observability. +- **databasev2** cannot observe table size without it + ([bounded-tables 5](../databasev2/05-bounded-tables-eviction.md)) and leans on + it for per-change benchmark CI + ([databasev2 story](../databasev2/00-story.md)); the RAM-ceiling study read + RSS from `/proc` by hand ([databasev2 1](../databasev2/01-ram-ceiling-measurement.md)) + precisely because this does not exist. +- The **rate limiter**'s ephemeral-row expiry is lazy "because porch has no + timer and iteration 30 owns" the sweep story (that "iteration 30" is now this + one — [porch 1](../porch/01-store-backed-middleware.md)). + +Fiber ships `expvar` and `pprof` as middleware; the equivalents here are runtime +work, because the numbers they expose (allocations, fiber counts, shard load, +GC pauses) live in the C runtime, not in `.wo`. + +## What it should deliver (scope to be refined) + +- **Runtime counters and gauges** — allocations, arena high-water, live fiber and + actor counts, per-shard load, GC pause totals, request counters — exposed + through one endpoint the app can mount. +- **A profiling story** — CPU and heap sampling, the `pprof` equivalent, so a hot + path can be found rather than guessed at. +- **A stack trace on trap** — today a trap is a 500 and a line; a trace at the + trap site is the cheapest debugging win and may be separable from the metrics + work. + +## Forks the brainstorm must settle + +1. **Counters only, or profiling too?** Counters and gauges are a bounded, mostly + `.wo`-plus-a-few-builtins surface; CPU/heap profiling needs sampling + machinery in the runtime and is a much larger commitment. Splitting profiling + into its own iteration is a legitimate outcome. +2. **Exposition format.** Prometheus text (the ops-standard, scrape-friendly), + an `expvar`-style JSON blob, or both. The format decides who can consume it + without a translator. +3. **Pull endpoint or push.** A mounted `/metrics` endpoint (pull) fits the + proxy-fronted, single-binary model; a push to a collector needs + `net.connect`, which does not exist (iteration 38) — so pull is almost + certainly the answer, but say so. +4. **Is stack-trace-on-trap in this iteration at all?** It is separable, it is + the highest debugging value per line, and it touches the trap path rather than + the metrics path — a candidate to land first and alone. + +## Out of scope (named, owned elsewhere) + +- **Per-change CI and fuzzing.** Frequently lumped under the old "iteration 30" + but they are tooling and process, not a runtime surface; they belong to a + CI/ops story, not this one. The benchmark *harness* already exists (language + iteration 22). +- **Distributed tracing / OpenTelemetry export.** Needs `net.connect` + (iteration 38) and a wire protocol; a later slice if a consumer appears. +- **Alerting, dashboards.** Downstream of exposition, not the runtime's job. + +## Info + +Consumers exist and are named above, so this is not a primitive shipped as +decoration. Ordering: stack-trace-on-trap has no dependency and could lead; +counters/gauges are next; profiling is the heaviest and most separable. Nothing +here depends on the porch track — the dependency runs the other way. diff --git a/docs/stories/runtime-v2/08-symmetric-cipher.md b/docs/stories/runtime-v2/08-symmetric-cipher.md new file mode 100644 index 0000000..7e117ea --- /dev/null +++ b/docs/stories/runtime-v2/08-symmetric-cipher.md @@ -0,0 +1,87 @@ +--- +track: runtime-v2 +iteration: "8" +status: pending +readiness: refine +--- + +# runtime-v2 8 — a symmetric cipher: authenticated encryption for cookies and data at rest + +> Created 2026-09-06 from the porch-vs-fiber scope-gap analysis +> ([exploration](../../plan/exploration/fiber/01-porch-vs-fiber-scope-gap.md)) as +> a runtime-v2 iteration — a hand-rolled cipher builtin with a `types.ml` row is +> exactly the track's builtin-sized-seam shape (the iteration 42 precedent). (It +> is not a language-track iteration; the language number 43 was already spent on +> the wmux foundation.) **`readiness: refine`** — the +> gap, its consumer and its forks are named; not brainstormed to `ready`. + +## Why this exists + +The runtime has digests only — SHA-1, SHA-256, HMAC-SHA256, base64 +([iteration 34](../language-runtime-database/34-crypto-builtins.md)) — and, from the porch track, +`random_bytes` ([porch 2](../porch/02-randomness-and-cookies.md)). Those let a +program *authenticate* and *sign* a value, and *mint* a random one. None of them +let it *encrypt* — turn a plaintext into a ciphertext only the key-holder can +read. + +That absence is a named porch limit: +[porch 2](../porch/02-randomness-and-cookies.md) scopes out **encrypted +cookies** explicitly — "Fiber's `encryptcookie` needs a symmetric cipher, and +the runtime has digests only. Signed-and-readable is honest and sufficient for a +session id; encrypting a payload is a separate ask with a separate primitive +behind it." This is that separate primitive. + +Signed-and-readable (what porch has) is correct for a session id — the client +may see it, it just may not forge it. Encryption is for the case where the +*payload itself* must be hidden from the client: an encrypted cookie carrying +app state, or a database field encrypted at rest. + +## What it should deliver (scope to be refined) + +- **An AEAD primitive** — authenticated encryption with associated data — as one + or two builtins in the crypto family beside `hmac_sha256`: encrypt (key, + nonce, associated-data, plaintext) → ciphertext+tag, and decrypt returning the + plaintext or nil on any authentication failure. AEAD, not a bare cipher, + because unauthenticated encryption is a footgun that ships. +- **The porch consumer**: an `encryptcookie`-equivalent — a cookie whose value is + encrypted, not merely signed — layered on iteration 2's cookie machinery. + +## Forks the brainstorm must settle + +1. **Which cipher? This is the load-bearing fork.** AES-256-GCM is what browsers, + fiber and every peer expect — but constant-time AES in pure software (no + AES-NI intrinsics) is genuinely hard to get right. ChaCha20-Poly1305 + (RFC 8439) is modern, is far easier to implement constant-time in portable C, + and is what a from-scratch no-dependency runtime should probably prefer — at + the cost of being less "expected." The runtime hand-rolls its crypto (the + SHA-256 precedent, no external dependency), which weighs toward ChaCha. +2. **Nonce management.** A reused nonce is catastrophic for both GCM and ChaCha. + Caller-supplied nonces put that footgun in every app; a builtin-generated + random nonce (drawing on iteration 2's `random_bytes`, prepended to the + ciphertext, as fiber's `NewGCMWithRandomNonce` does) removes it. Leaning + builtin-generated — so this iteration is ordered after porch 2's builtin. +3. **Key handling.** A raw 32-byte key (from `random_bytes`, carried as base64 in + config, the `encryptcookie.GenerateKey` shape) with a length check, versus a + passphrase-plus-KDF. Leaning raw key with validation; a KDF is its own ask. +4. **Hand-roll versus vendor.** Doctrine is no external dependencies. A + hand-rolled ChaCha20-Poly1305 in C is bounded and well-specified; hand-rolled + AES-GCM is more error-prone. This fork is the practical face of fork 1. + +## Out of scope + +- **Asymmetric crypto** (RSA, ECDH, signatures beyond HMAC). A different, much + larger surface with no current consumer. +- **Key rotation, a KMS, envelope encryption.** Operational key management is its + own story if a consumer appears. +- **TLS.** Proxy-terminated by doctrine; this cipher is for application payloads, + not the transport. +- **Compression before encryption** (the CRIME/BREACH interaction). A caller + concern to document, not a primitive. + +## Info + +One named consumer today (encrypted cookies), with database-field-at-rest as a +plausible second — enough to not be decoration, not so much as to over-build. +Depends on iteration 2's `random_bytes` (for the nonce, fork 2) and extends +iteration 34's crypto builtins. Pure compute — no actors, not exposed to the +lang-41 hang. diff --git a/docs/stories/runtime-v2/09-in-process-tls.md b/docs/stories/runtime-v2/09-in-process-tls.md new file mode 100644 index 0000000..857f373 --- /dev/null +++ b/docs/stories/runtime-v2/09-in-process-tls.md @@ -0,0 +1,117 @@ +--- +track: runtime-v2 +iteration: "9" +status: pending +readiness: refine +--- + +# runtime-v2 9 — in-process TLS: retiring the proxy-termination doctrine + +> Created 2026-09-07 from the gap [`jarvis`](../jarvis/00-story.md) surfaces — an +> assistant must dial an LLM over HTTPS, and the runtime has no outbound TLS. The +> developer chose the **full overturn**: the runtime gains TLS **both +> directions**, and the standing "TLS is the proxy's job" doctrine is retired. +> **`readiness: refine`** — the gap, its consumers and its forks are named here; +> the implementation is deliberately left as this story's load-bearing fork, not +> settled. + +## Why this exists — and what it overturns + +Three documents record the same standing decision, and this story reverses it: + +- *"TLS — permanently the proxy's job (framework doctrine)"* — + [language 34](../language-runtime-database/34-crypto-builtins.md) (crypto + builtins, line ~83). +- *"TLS — proxy-terminated, by doctrine, unchanged… the story says so out loud + rather than implying HTTPS clients"* — + [language 38](../language-runtime-database/38-content-platform-capabilities.md) + (which adds `net.connect` as **plaintext** outbound TCP and explicitly refuses + HTTPS). +- *"TLS, HTTP/2 | nobody — proxy-terminated by doctrine"* — + [porch](../porch/00-story.md)'s "what this track does NOT own". + +The doctrine was reasonable while nothing in-tree needed to *dial* anything: a +front proxy terminates inbound TLS, and there were no outbound callers. jarvis +breaks that — its whole job is to reach a remote API — and the developer's +direct-HTTPS choice for it means the runtime, not a companion, owns the +connection. Rather than carve out a one-directional exception, the decision is to +give the runtime TLS in **both** directions: outbound so a `.wo` program can dial +HTTPS, and inbound so porch can terminate TLS itself instead of mandating a +proxy in front of every deployment. + +This is **not a builtin-sized seam** like the rest of this track. TLS 1.3 plus +X.509 certificate validation is a large, security-critical subsystem — the one +place the runtime's hand-roll-everything habit (the sha256 precedent) should not +be assumed to extend. That tension is the load-bearing fork below. + +## What it should deliver (scope to be refined) + +- **Outbound TLS client** — a `.wo` program dials an HTTPS endpoint: a TLS + handshake over the TCP socket `net.connect` (language 38) provides, with server + certificate validation against a trust store. jarvis's direct path, and + language 38's deliberately-excluded HTTPS half. +- **Inbound TLS server** — porch terminates TLS on its own listener (cert + key + loaded at startup), retiring the "put a proxy in front" requirement for a + single-binary deployment. +- **Certificate validation and a trust store** — X.509 chain verification, + hostname/SNI checks outbound; certificate + private-key loading inbound. This + is where most of the risk and most of the code live. + +## Forks the brainstorm must settle + +1. **Implementation source — the load-bearing fork (left open by direction).** + Vendor a small, audited TLS library (mbedTLS or BearSSL) compiled into the + single static binary — correct and maintainable, but a build-time external + dependency, an explicit exception to the no-external-deps doctrine the runtime + otherwise holds — versus hand-rolling a TLS 1.3 subset plus X.509 in C, which + matches the sha256 precedent but is thousands of security-critical lines and a + hand-rolled certificate validator is a CVE factory (strongly discouraged). The + honest lean is vendor-a-lib; TLS is exactly the thing not to hand-roll. This + fork decides whether the whole "single static binary, no external deps" story + gains a footnote. +2. **Phasing.** Outbound first (jarvis's actual need) with inbound to follow, or + both together since the handshake machinery and the vendored library are + shared and only the client-vs-server role and validation direction differ. +3. **Trust store and cert provisioning.** Where the outbound trust anchors come + from (the system CA bundle, and its path across distros), and how the inbound + side is handed its certificate and key (files, env, a reload story). +4. **TLS version and cipher policy.** TLS 1.3 only (simplest, modern, smallest + attack surface) versus 1.2+1.3 (broader reach). Leaning 1.3-only. + +## Consumers + +Named, so this is not a capability shipped as decoration: + +- **[jarvis 1](../jarvis/00-story.md)** — outbound HTTPS to the LLM API (the + reason this story exists). +- **porch** — inbound TLS termination, retiring the mandatory front proxy for a + single-binary deployment. +- **language 38** — the outbound HTTPS half it excluded by doctrine; this story + is where that exclusion is lifted. + +## Dependencies + +- **[language 38](../language-runtime-database/38-content-platform-capabilities.md)** + — `net.connect` (outbound TCP) is the socket the outbound handshake runs over; + the client half of this story sits directly on it. + +## Out of scope + +- **HTTP/2.** A separate protocol concern, parked behind language iteration 23 + regardless; TLS is its prerequisite, not its owner. +- **Mutual TLS / client certificates.** A later slice if a consumer asks; the + first cut authenticates the server, not the client. +- **Updating the doctrine documents.** Retiring "TLS is the proxy's job" means + correcting [language 34](../language-runtime-database/34-crypto-builtins.md), + [language 38](../language-runtime-database/38-content-platform-capabilities.md) + and [porch](../porch/00-story.md) when this lands — a follow-up bookkeeping + pass, named here so it is not forgotten, not part of the runtime work. + +## Info + +This is the heaviest iteration in the runtime-v2 track and the only one that +forces a doctrine reversal and, most likely, an external-dependency exception — +both flagged above rather than buried. It is pure I/O-plane work (a handshake +layer over the existing socket verbs); no actors, so it is not exposed to the +lang-41 hang. It gates jarvis entirely: until it lands, jarvis cannot reach a +model at all.