diff --git a/docs/00-dependency-graph.md b/docs/00-dependency-graph.md index e412984..8122190 100644 --- a/docs/00-dependency-graph.md +++ b/docs/00-dependency-graph.md @@ -21,6 +21,7 @@ flowchart TD classDef parked fill:#6e7781,color:#fff,stroke:none classDef specd fill:#0969da,color:#fff,stroke:none classDef open fill:#eac54f,color:#000,stroke:none + classDef inprog fill:#8250df,color:#fff,stroke:none FOUND["1–6 foundation: doctrine, VM, compiler, binary, surface, stdlib"]:::done I7["7 log-watcher proof"]:::done @@ -38,7 +39,7 @@ flowchart TD I36["36 operator parity: not/bitwise/hex literals β€” code landed 2026-08-22, awaiting the manual pass"]:::specd RELEASE["packaging + release pipeline βœ… 2026-08-25 (no story: VERSION, just dist, install-accept, release.yml)"]:::done - I18["18 framework v2: transaction{} + cache/flags/jobs (⏸ hold 2026-08-21; spec+plan approved, held intact)"]:::parked + I18["18 transaction{} πŸ”„ hold lifted + split 2026-09-11 (cache/flags/jobs β†’ porch 10, graph 4); T1–T6, T8, T9 landed same day, corpus green; open: kill -9 battery (T7)"]:::inprog I9c["20 cross-program tables (⏸ hold 2026-08-21; channel half-built)"]:::parked I9d["21 keypair attach auth (⏸ hold 2026-08-21; crypto floor now exists via 34)"]:::parked @@ -47,8 +48,8 @@ flowchart TD I24["24 chat + actor lifecycle πŸ”„ THE LIVE SLICE (absorbing 31 + 34)"]:::specd I34["34 crypto builtins βœ… code landed as 24's T1 (ids 85-87)"]:::done I31["31 actor lifecycle β€” call/mailbox-cap/death landed in 24; monitor + time.after (ids 89/90) open"]:::specd - I9f["23 io_uring group-commit"]:::open - I32["32 WAL checkpoint (append-only today; bounds replay)"]:::open + I9f["23 io_uring group-commit βœ… part A 2026-08-28 as databasev2 4 (part B refine)"]:::done + I32["32 WAL checkpoint βœ… 2026-08-29 as databasev2 3 (compaction by rewrite + rename)"]:::done I33["33 single-file store WO_DATA=.db (driver-only, off-chain)"]:::open I25["25 HTTP service layer β€” `service` blocks (⏸ hold 2026-08-21; story file removed, plan remains)"]:::parked I11["11 fibers βœ… 2026-08-21"]:::done @@ -92,7 +93,8 @@ flowchart TD I11 --> I24 I35 --> I24 I31 --- I24 - I9f --> I32 + %% 23 and 32 compose on the WAL commit path; neither needs the other (databasev2 story, corrected 2026-08-29) + I9f --- I32 I32 --- I33 I9 --> I26 I25 --> I26 @@ -109,13 +111,15 @@ flowchart TD ``` Reading it: **the live slice is 24** (chat + actor lifecycle, absorbing 31 -and 34), and the chain behind it is 23 β†’ 32. Everything else with all-green +and 34), and the chain behind it, 23 β†’ 32, is done (databasev2 4 part A +2026-08-28, databasev2 3 2026-08-29). Everything else with all-green incoming arrows is startable: **33** (driver-only, off-chain), **38** (the fs-mutation and outbound-socket gaps), and **30**'s remaining half (per-change CI and fuzzing β€” the release pipeline covered only publishing). **36** needs no work, only the developer's manual pass over -`docs/examples/operators/`. The held tail β€” 18, 20/21, 25, 26, 27, 28, 29 β€” -resumes on its own precedence notes; 29 and what is left of the drain still +`docs/examples/operators/`. The held tail β€” 20/21, 25, 26, 27, 28, 29 β€” (18's +hold lifted 2026-09-11, above) resumes on its own precedence notes; 29 and +what is left of the drain still sit behind 26 by the 2026-08-08 scope directive (dashed), not by any technical edge. Note what left the drain: `pub(read)`, `using` and `#if` all shipped, so only the WO-E225/ADT rosters and group-by aggregates remain in @@ -135,7 +139,7 @@ flowchart TD I7b2["7b per-shard collector (done β€” the precondition 8 waited on)"]:::rt I8x["8 shard-actor runtime: thread-per-core, ownership-move messages"]:::rt - I9fx["23 io_uring group-commit (batch = the shard tick)"]:::rt + I9fx["23 io_uring group-commit (batch = queue drain) βœ… part A 2026-08-28"]:::done I11x["11 fibers: reduction-budget preemption, blocking builtins park"]:::rt I9ex["22 baseline (numbers 8/23 sign against)"]:::rt @@ -150,7 +154,7 @@ flowchart TD FIBJOBS2["fiber-scheduled jobs (replaces drain-on-request; queue table stays)"]:::v2 CANCELRB["cancellation β†’ transaction rollback"]:::v2 - I18x["18 transaction{} + jobs"]:::v2 + I18x["18 transaction{} (jobs moved to porch 10, 2026-09-11 β€” graph 4)"]:::v2 I7b2 --> I8x I9ex --> I9fx @@ -246,31 +250,32 @@ HS256 (the hard stop). Still gated: timeouts/unix-socket/peer-verify Note: 21's keypair crypto is its own C implementation (already on branch `keypair-auth`) β€” it neither waits for nor feeds this chain. -## 4. Framework v2 (iteration 18) β€” internal order +## 4. Language 18 β€” `transaction { }` (was "Framework v2"; split 2026-09-11) + +**Redrawn 2026-09-11.** The hold lifted (developer: "implement language 18") +and codd-shoney re-settled the scope: 18 is now the engine + language block +alone. `cache.wo`, `flags.wo`, `jobs.wo` and the transactional demo moved to +[porch 10](stories/porch/10-memory-features-over-table.md) (`refine`, stub), +which needs 18's `transaction { }` for its jobs demo and porch 1–3 otherwise. ```mermaid flowchart TD classDef piece fill:#0969da,color:#fff,stroke:none - classDef indep fill:#1a7f37,color:#fff,stroke:none - classDef later fill:#eac54f,color:#000,stroke:none + classDef inprog fill:#8250df,color:#fff,stroke:none + classDef refine fill:#eac54f,color:#000,stroke:none - TXN["transaction{} (engine undo log + language block)"]:::piece - JOBS["jobs.wo: wf_jobs + enqueue + JobRunner + idle() drain"]:::piece - DEMO["web-app demo: transactional order+confirm, GET /jobs, flags route"]:::piece - CACHE["cache.wo: TTL + FIFO"]:::indep - FLAGS["flags.wo: wf_flags + read-through map"]:::indep - TPRMW["txn-per-request middleware (v1 ledger's storage row; single-thread OK)"]:::later + TXN["language 18: transaction{} β€” compiler block, kind-6 log record, engine undo list, VM re-raise: landed 2026-09-11 (T1–T6); open: kill -9 battery"]:::inprog + TPRMW["txn-per-request middleware (v1 ledger's storage row; single-thread OK)"]:::piece + P10["porch 10: cache.wo, flags.wo, jobs.wo + the transactional demo (stub, refine)"]:::refine - TXN --> JOBS - JOBS --> DEMO - FLAGS --> DEMO TXN --> TPRMW + TXN -. moved 2026-09-11 .-> P10 ``` -Cache and flags are dependency-free warm-ups; `transaction { }` is the -critical path (the only engine + language work); jobs compose on it; the -demo and gate close it. Fiber-scheduled jobs and cancellationβ†’rollback -appear in graph 2 β€” they need iteration 11 as well as 18. +`transaction { }` is the only engine + language work left in this iteration; +everything that only needed the WAL's staged batch as a `.wo` consumer moved +downstream to the track that owns `.wo` product code. Fiber-scheduled jobs and +cancellationβ†’rollback appear in graph 2 β€” they need iteration 11 as well as 18. ## 5. porch β€” the web framework track @@ -278,7 +283,10 @@ States live on [the board's porch section](stories/00-status.md). **The whole track (2–8) is `readiness: ready`** as of the 2026-09-06 brainstorm; **1** is done, **9** is held (blocked on the lang-41 arena hang, not an enhancement). Three independent roots: **2** (the auth chain), **6** (the streaming chain), -**5** (anytime, no incoming edges at all β€” not even iteration 2). +**5** (anytime, no incoming edges at all β€” not even iteration 2). **10** is a +`refine` stub added 2026-09-11 (language 18's split β€” TTL cache, `@table` +feature flags, durable job queue) and is not part of the "whole track ready" +count. This graph makes the **cross-track language edges** visible: the three builtins the track needs, each drawn as a `lang` node feeding the story that owns it. @@ -289,8 +297,10 @@ flowchart TD classDef ready fill:#0969da,color:#fff,stroke:none classDef held fill:#6e7781,color:#fff,stroke:none classDef lang fill:#8250df,color:#fff,stroke:none + classDef refine fill:#eac54f,color:#000,stroke:none - RB["language work: random_bytes builtin (bare-name, id 84/90) β€” porch 2 Phase A"]:::lang + TXN["language 18: transaction{} (T1 in flight)"]:::lang + RB["random_bytes builtin βœ… 2026-09-09 (bare-name, id 119 β€” one shared enum with wob.h/loader arity) β€” porch 2 Phase A"]:::done DFL["language work: deflate + crc32 builtins (C) β€” porch 7 Phase C"]:::lang TU["language work: time.utc builtin (gmtime sibling of time.local) β€” porch 8 Phase A"]:::lang @@ -303,6 +313,7 @@ flowchart TD P7["porch 7 SSE + compression"]:::ready P8["porch 8 static files + lifecycle"]:::ready P9["porch 9 idempotent replay (ready β€” unblocked 2026-09-09)"]:::ready + P10["porch 10 memory features over @table: cache/flags/jobs (stub, refine β€” split from language 18, 2026-09-11)"]:::refine L41["language 41 actor-arena double free βœ… fixed 63065ff (cross-shard marshal)"]:::done L44["language 44 poison-on-free βœ… (41's decision 3: a freed header can never pass for live; double free aborts)"]:::done L41 -.follow-up.-> L44 @@ -319,6 +330,10 @@ flowchart TD TU --> P8 L41 -.fixed 2026-09-09 β€” no longer blocks.-> P9 P1 -.re-scope 79e6da4: replay-on-retry split out of 1.-> P9 + TXN --> P10 + P1 --> P10 + P2 --> P10 + P3 --> P10 ``` Edges corrected by the 2026-09-06 brainstorm: `P5 --> P7` (gzip's @@ -327,7 +342,9 @@ Edges corrected by the 2026-09-06 brainstorm: `P5 --> P7` (gzip's (iteration 5's shape), not iteration 2's repeated-header work. `P5 --> P8` is the `Download`/`Attachment` helper. The three `lang` nodes are the track's entire language bill; each is a builtin with a named consumer, none shipped as -decoration. +decoration. **10** (added 2026-09-11) needs language 18's `transaction { }` +for its jobs demo and 1–3 for the store pattern, session-keyed cache and +flags read-through β€” see graph 4 for 18's own state. ## 5a. porch's language-driven gaps (out-of-scope features and the language stories that own them) @@ -342,12 +359,13 @@ flowchart LR classDef refine fill:#eac54f,color:#000,stroke:none classDef held fill:#6e7781,color:#fff,stroke:none classDef gap fill:#cf222e,color:#fff,stroke:none + classDef inprog fill:#8250df,color:#fff,stroke:none L29["language 29 @derive (⏸ hold)"]:::held L38["language 38 net.connect βœ… landed (id 110); proxy middleware now buildable"]:::done L43["runtime-v2 8 symmetric cipher (refine, NEW 2026-09-06)"]:::refine L30["runtime-v2 7 observability (refine, moved from language 30, 2026-09-06)"]:::refine - L18["language 18 TTL cache + transaction{} (⏸ hold)"]:::held + L18["language 18 transaction{} (hold lifted 2026-09-11; T1 in flight) β€” TTL cache moved to porch 10"]:::inprog L31["language 31 cancellation βœ… (landed in 24)"]:::done BIND["typed request binding (fiber Bind)"]:::gap @@ -369,8 +387,9 @@ flowchart LR backpressure are unblocked at the language level and wait only on a porch slice to consume them. The other five gaps are gated on an upstream story: two brand-new runtime-v2 iterations (8 cipher, 7 observability β€” moved out of the -language track 2026-09-06), two language iterations on hold (29, 18), one pending -a spec (38). Landed enablers the +language track 2026-09-06), one language iteration on hold (29) and one +unheld and in flight (18, since 2026-09-11 β€” its TTL-cache half of `CACHE` now +lives in porch 10), one pending a spec (38). Landed enablers the track already consumed β€” 34 (crypto digests), 36 (bit operators), 35 (net seams) β€” are green in graphs 1–3 and not repeated here. @@ -402,8 +421,60 @@ flowchart TD GSIG["runtime-v2 3 βœ… 2026-09-02 signals as events: signal.on delivers Signal records"]:::done GTERMIOS["runtime-v2 4 βœ… 2026-09-02 termios: raw/restore, restore a runtime obligation"]:::done GFDPASS["runtime-v2 5 βœ… 2026-09-02 fd passing: send_fd/recv_fd/connect_unix"]:::done - GVTE["VTE grid in pure .wo + unicode width tables (pinned against recorded sessions)"]:::gap - WMUX["wmux 1 (was language 43): server owns sessions/PTYs in durable tables, thin client hands over its tty β€” reattach after server RESTART replays from the WAL"]:::product + GVTE["VTE grid in pure .wo + unicode width tables βœ… (rung 2; UTF-8 decode + term.width landed)"]:::done + DB2W["databasev2 2 per-table durable/resident βœ… 2026-09-10 β€” durable default + WAL wmux 1 persists sessions/scrollback into"]:::done + WMUX["wmux 1 foundation βœ… 2026-09-02 (was language 43): server owns sessions/PTYs in durable tables, thin client hands over its tty β€” reattach after server RESTART replays from the WAL"]:::done + W2["wmux 2 the screen βœ… VTE grid"]:::done + W3["wmux 3 windows + status βœ…"]:::done + W4["wmux 4 split panes βœ… (2-pane vertical)"]:::done + W5["wmux 5 copy mode βœ…"]:::done + W6["wmux 6 multi-client βœ… (mirroring)"]:::done + W7["wmux 7 command system βœ…"]:::done + W8["wmux 8 hooks + control βœ…"]:::done + W9["wmux 9 parity audit βœ…"]:::done + W10["wmux 10 layout tree β€” 🟑 first slice DONE 2026-09-03: horizontal split-window -h, select-pane -L/R/U/D, zoom; 2-pane max, N-way/swap/break/persistence pending"]:::gap + W11["wmux 11 formats + options + key rebinding βœ… 2026-09-02 β€” durable options/binds, #{...} status format, one run_command dispatcher; folded rung 14's prompt-race fix; fixed a PTY-EIO reader spin + a kill-session chunk race. gate 36/0"]:::done + W12["wmux 12 resize + mouse β€” 🟑 first slice DONE 2026-09-03/04: attach-time term.size sizing + SGR mouse (wheel/click/status-row); live SIGWINCH + min-size pending (rt2 3/6 ready)"]:::gap + W13["wmux 13 copy selection + search β€” 🟑 first slice DONE 2026-09-03: char-range vi v/y yank β†’ buffer+OSC52; only search + rectangle pending"]:::gap + W14["wmux 14 control surface (prompt-race fix DONE in rung 11; narrows to control-mode commands + %notifications)"]:::gap + W15["wmux 15 terminfo β€” 🟑 terminfo-lite DONE 2026-09-03: a TERM allowlist retired the foreign-TERM refusal; full compiled-terminfo parsing pending"]:::gap + W16["wmux 16 durability polish (pane persistence, killw compaction) β€” 🟑 first slice DONE 2026-09-02: Window owns+reaps its panes (spawns in-actor so wait_dl works on its shard), zombie leak fixed, gate 37/0"]:::gap + W18["wmux 18 key tables (new, from the config audit) β€” 🟑 first slice DONE 2026-09-03: no-prefix RootBind table, Meta/named key_code, tty key decoder in Input; bind-key -n works; gate 42/0. copy-mode-vi + -r repeat pending. Also landed: dynamic sizing (term.size), alt-screen, erase 0/1, SGR reset, UTF-8 decode, O(n log n) replay"]:::gap + W19["wmux 19 mouse-driven UX (new) β€” 🟑 active-pane border + status-row clickβ†’window DONE 2026-09-04; drag-resize/drag-select pending. Forks open (scope split, motion mode, drag owner)"]:::gap + W17["wmux 17 formats v2 βœ… 2026-09-03 β€” #(shell) cached+timer, recursive #{...} conditionals/modifiers, #{time}/#{host_short}/#{window_name}"]:::done + W20["wmux 20 display-popup βœ… 2026-09-03/04 β€” session-owned modal float, -B borderless, rounded border, popup wheel forward; drove the OSC-swallow + frame-coalesce VTE fixes"]:::done + W21["wmux 21 sesh + switch-client βœ… 2026-09-04 β€” in-session switch-client -t/-l, reg threaded into sessions, sync call hand-off (fds move without close, B spawns a fresh Input, old Input exits on success), B-occupied refuses. Fixed a ?actor nullable schema-reorder. Gate 54/0"]:::done + W22["wmux 22 theming βœ… 2026-09-04 β€” style_sgr engine (fg/bg/attrs from durable options), active-pane border marker, automatic-rename via OSC title"]:::done + W23["wmux 23 plugin ports β€” thumbs/fzf/fzf-url via capture-pane + a popup picker (port vs tmux-compat shim). Forks open"]:::gap + W11 --> W18 + W12 --> W19 + W13 --> W19 + W10 -.drag-resize only.-> W19 + W11 --> W17 + W2 --> W20 + W6 --> W21 + W20 --> W21 + W11 --> W22 + W10 --> W22 + W20 --> W23 + W12 --> W23 + WMUX --> W2 + W2 --> W3 + W3 --> W4 + W4 --> W5 + W5 --> W6 + W6 --> W7 + W7 --> W8 + W8 --> W9 + W9 --> W10 + W4 --> W10 + W7 --> W11 + W6 --> W12 + W5 --> W13 + W8 --> W14 + W1TERM["(rung 1 fixed-profile refusal)"]:::done + W1TERM -.retired by.-> W15 + W10 --> W16 TINFO["terminfo fork: parse the db in .wo vs fixed xterm-256color + refusal by name (decide at 43's brainstorm)"]:::later TMONO["time.mono returns (status clock, repaint pacing) β€” v2"]:::later @@ -414,16 +485,19 @@ flowchart TD GTERMIOS --> WMUX GFDPASS --> WMUX GVTE --> WMUX + DB2W --> WMUX TINFO -.settled at wmux's brainstorm.-> WMUX TMONO -.v2.-> WMUX ``` **The track landed whole on 2026-09-02** β€” every runtime edge into wmux -is green; what remains for wmux 1 is its own `.wo` work (the VTE grid + -unicode width node) and its brainstorm's terminfo fork. Sibling reuse: +is green. The VTE grid + unicode-width node landed (rung 2), and the ladder +is now through rung 22 (see the wmux table on the board); rungs 10/12/13/15 +have first slices, rung 23 (plugin ports + a tmux-compat CLI) remains the +big open item. Sibling reuse: the alacritty Wayland stage reuses GFDPASS + GVTE; the zen CDP driver now lacks only a WebSocket client; skillhost (28) has its stdin -transport. +transport. One edge added 2026-09-10: [databasev2 2](stories/databasev2/02-table-storage-modes.md) β†’ wmux 1, drawn above as `DB2W`, because wmux 1 persists sessions/scrollback in durable `@table` classes and replays from the WAL on reattach β€” the dependency the prose already named without a node. ## 7. jarvis β€” the AI-assistant track and everything it waits on @@ -501,6 +575,83 @@ flowchart TD 4, 8, 9 to complete porch) β†’ **jarvis 1**. Nothing on the runtime side is outstanding; every remaining edge into jarvis 1 is a porch iteration. +## 8. databasev2 β€” the database beyond RAM + +The third track ([databasev2](stories/databasev2/00-story.md)): what happens +when the data does not fit in memory. Its 3 and 4 are the language track's 32 +and 23 renumbered β€” graph 1 still carries them as `I32`/`I9f`, green since +2026-08-29/28. Arrows point AT the iteration that needs the other, as in the +track's own ASCII graph; two edges are undirected: 3–4, which compose on the +WAL commit path and need each other in neither direction, and 2–3 (added +2026-09-10 β€” this graph had dropped it; the story's own graph always carried +it, [00-story.md:169-171](stories/databasev2/00-story.md)) β€” 2 needs 3's +offset map to survive compaction, and 3 has called 2's row API since task 5d, +so the coupling runs both ways. 9 and 10 (cross-program tables, keypair +attach) are held and not drawn. + +```mermaid +flowchart TD + classDef done fill:#1a7f37,color:#fff,stroke:none + classDef inprog fill:#8250df,color:#fff,stroke:none + classDef ready fill:#0969da,color:#fff,stroke:none + classDef refine fill:#eac54f,color:#000,stroke:none + classDef hold fill:#6e7781,color:#fff,stroke:none + + D1["databasev2 1 RAM ceiling measured βœ… 2026-08-27"]:::done + D2["databasev2 2 per-table durable/resident βœ… 2026-09-10 β€” 6a: refuse durable:true without WO_DATA, WO_EPHEMERAL=1 escape, .wob v8 table bit"]:::done + D3["databasev2 3 WAL checkpoint βœ… 2026-08-29 (was 32)"]:::done + D4["databasev2 4 group commit πŸ”„ β€” part A βœ… 2026-08-28; part B re-brainstormed 2026-09-10, GO measured (forks 6/7), fold pending β€” refine (was 23)"]:::inprog + D5["databasev2 5 bounded tables + eviction β€” ready 2026-09-10 (12 forks, review_pending), status pending; Phase A is the resident byte budget moved from 2 (2026-09-09)"]:::ready + D6["databasev2 6 cold tiering β€” hold (superseded by 2's resident: keys)"]:::hold + D7["databasev2 7 single-file store βœ… 2026-09-10 β€” WO_DATA=.db (was 33)"]:::done + D8["databasev2 8 query grammar from corpora β€” hold, refine (count landed 2026-08-16; exists open) (was 27)"]:::hold + D11["databasev2 11 bounded delta chains βœ… 2026-08-30"]:::done + D12["databasev2 12 schema migrations βœ… 2026-08-31"]:::done + D13["databasev2 13 fresh-log keys-resident seed SEGV βœ… fixed 2026-09-10"]:::done + + P1["porch 1 store-backed middleware βœ… 2026-08-30"]:::done + P2["porch 2 randomness + cookies (ready)"]:::ready + P3["porch 3 sessions (ready)"]:::ready + + D1 -- budget default follows the measurement --> D5 + D2 -- the byte budget, moved 2026-09-09 --> D5 + D2 --> D11 + D2 --> D12 + D3 --- D4 + D3 --- D2 + D2 -- volatile tables; store.wo is default-durable today, so fork 6 makes WO_DATA or WO_EPHEMERAL=1 whole-program --> P1 + D2 --> P2 + D2 --> P3 + D2 -- the offset map D13's fix touches --> D13 + D12 -- the schema head record D13's fix touches --> D13 +``` + +Node D13, added 2026-09-10, fixed the same day: a defect found while smoking +databasev2 7, not that iteration's fault (it reproduced identically in the +pre-existing directory form). It needed 2 (the keys-resident offset map) and +12 (the schema head record) β€” both are the mechanism the crash lived in. +Fixed by `6310078` (`wo_wal_next_offset` stages the pending schema head +before returning an offset) + `1b6750d` (NULL-`msg` guard in +`wo_wal_fold_row_at`); `just residency` 32/0. The separate +`residency.keys.fit` rc 74 bug (compaction/replay of keys-resident offsets) +is **not** the same defect and stays open under codd.md's "Next bugs". + +**States as of 2026-09-10** (the board carries the words; this is the glance): + +| databasev2 | status / readiness | what is left | +| --- | --- | --- | +| 1 RAM ceiling | βœ… done | β€” | +| 2 per-table storage | βœ… done (2026-09-10) | β€” (6a landed with the `.wob` v8 table bit; 6b lives in 5) | +| 3 WAL checkpoint | βœ… done | β€” | +| 4 group commit | πŸ”„ in-progress / refine | part B re-brainstormed 2026-09-10 (GO measured, forks 6/7); fold into the story, `.dev/zack/databasev2-4b.md` | +| 5 bounded tables | ⬜ pending / **ready** (2026-09-10) | developer review of the twelve `review_pending` forks, or a prebuild brief for Phase B; Phase A is startable now | +| 6 cold tiering | ⏸ hold / refine | superseded by 2; revisit only on a measurement | +| 7 single-file store | βœ… done (2026-09-10) | β€” (`WO_DATA` is a path, never a sentinel; `review_pending` developer second review) | +| 8 query grammar | ⏸ hold / refine | `count` landed; `exists` waits for a corpus | +| 11 bounded delta chains | βœ… done | β€” | +| 12 schema migrations | βœ… done | β€” | +| 13 fresh-log keys-resident seed SEGV | βœ… done (2026-09-10) | β€” (`residency.keys.fit` rc 74 is a separate, still-open bug) | + ## Maintenance rule When an iteration or slice lands, update its node's class here in the diff --git a/docs/stories/00-status.md b/docs/stories/00-status.md index 7773a6f..75e43dc 100644 --- a/docs/stories/00-status.md +++ b/docs/stories/00-status.md @@ -1284,7 +1284,7 @@ that sequences its tasks. Read one, approve, then the next starts. | 15 | [deps: `wo.toml [deps]`](language-runtime-database/15-deps-package-manager.md) | βœ… **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 | | 16 | [web framework](language-runtime-database/16-web-framework.md) | βœ… **landed 2026-08-19** β€” writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (Β§C) behind 8/23/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 | | 17 | [library projects + `internal/`](language-runtime-database/17-library-projects-internal.md) | βœ… **landed 2026-08-20** β€” `kind = "library"` in `wo.toml` (default `program`, so every existing manifest is byte-identical; unknown value = WO-E109 exit 2); `woc ` on a library runs the FULL pipeline entry-less and writes nothing, retiring iteration 16's `--emit` workaround; the no-entry build error names the kind; lib+bin dual works. Go's `internal/` rule as **WO-E108** at the consumer's own `use`, dep-boundary-only β€” the library imports its own interior freely. Framework reorganized: `internal/{parse,serve}.wo` behind the line, `http/form.wo` split out to keep `media_type`/`form_values` public. Driver-only change; VM/`.wob`/GC untouched. `just web-app` **26/0** (3 new checks), every standing gate unchanged | -| 18 | [framework v2: memory-rich features](language-runtime-database/18-memory-db-features.md) | ⏸ hold (2026-08-21); spec approved + plan authored, both held intact ([spec](../superpowers/specs/2026-08-20-memory-db-features-design.md), [plan](../superpowers/plans/2026-08-20-framework-v2-memory-features.md)): TTL cache + @table flags + durable job queue (drain-on-request) + `transaction { }` over the WAL's staged batch; pub/sub rejection expired with the arc (8/11 landed 2026-08-21) β€” revisit on unhold | +| 18 | [`transaction { }`](language-runtime-database/18-memory-db-features.md) | πŸ”„ **hold lifted 2026-09-11** (developer: "implement language 18"); re-settled and **split** β€” 18 keeps `transaction { }` only, TTL cache + `@table` flags + durable job queue moved to [porch 10](porch/10-memory-features-over-table.md) (`refine`, stub); `status: in-progress`, `readiness: ready`, five forks (3c/3d/3h/3j/3l) `review_pending`; zack on T1 (compiler surface) | --- @@ -1295,8 +1295,11 @@ that sequences its tasks. Read one, approve, then the next starts. | Language | πŸ”„ [iteration 36 β€” operator parity](language-runtime-database/36-operator-parity.md): `not`, bitwise `& \| ^ << >>`, hex/binary/`_` literals, compound assigns β€” CODE LANDED 2026-08-22 (branch operator-parity, `.wob` v6, all gates green; reference project `.dev/reference/go` drove the design). Awaiting the developer's MANUAL pass on `docs/examples/operators/` (no test fixtures by directive); unblocks story 34's pure-`.wo` HMAC question | [plan](../superpowers/plans/2026-08-22-operator-parity.md) | | Language | the framework v1-polish slice landed 2026-08-20 (branch framework-v1, awaiting merge); next per the order: brainstorm 20/21's forks | [order](#implementation-order-re-sequenced-2026-08-21--concurrency-chain) | | Runtime | βœ… **iteration 35 landed 2026-08-23** (branch `framework-v1b`, with framework v1 slice 2 + the serving slice): net deadlines/unix/peer (ids 91–95), fiber pooling, serve_conn + web-app fiber-per-connection β€” web-app gate 41/0, both WO_IO backends | [design](../superpowers/specs/2026-08-23-net-seams-park-design.md) | +| databasev2 | πŸ”„ [iteration 4 β€” group commit](databasev2/04-io-uring-commit.md): part A landed 2026-08-28; part B re-brainstormed 2026-09-10 (forks 1–5/8–10 `review_pending`, forks 6/7 settled in substance β€” GO measured β€” but fold pending, `.dev/zack/databasev2-4b.md`); `readiness: refine` until folded | [spec](../superpowers/specs/2026-08-28-wal-group-commit-design.md) | +| databasev2 | ⬜ [iteration 5 β€” bounded tables and eviction](databasev2/05-bounded-tables-eviction.md): brainstormed to `readiness: ready` 2026-09-10 (twelve forks, `review_pending`); `status: pending` β€” not started | [databasev2 5](databasev2/05-bounded-tables-eviction.md) | +| Language | πŸ”„ [iteration 18 β€” `transaction { }`](language-runtime-database/18-memory-db-features.md): hold lifted 2026-09-11, re-settled and split (TTL cache/flags/jobs moved to [porch 10](porch/10-memory-features-over-table.md), `refine`); zack on T1 (compiler surface); `readiness: ready`, five forks (3c/3d/3h/3j/3l) `review_pending` | [18](language-runtime-database/18-memory-db-features.md) | -**No slice is active.** Iteration 24 landed 2026-08-27 and its marker doc was +**Active slice: [language 18](language-runtime-database/18-memory-db-features.md) T1 (compiler surface) is in flight (hold lifted 2026-09-11), and [databasev2 5](databasev2/05-bounded-tables-eviction.md) Phase A is `ready` and startable (brainstormed 2026-09-10, per codd-pm's ranking) β€” see the standup entry above.** Iteration 24 landed 2026-08-27 and its marker doc was deleted per the convention. Everything pending is the concurrency chain (see *Pending* below) β€” **the chain's next link is [databasev2 4](databasev2/04-io-uring-commit.md)** (chain 5, the io_uring @@ -1541,17 +1544,18 @@ the language arc as v1 history. | # | Iteration | State | | --- | --- | --- | | 1 | [RAM ceiling: measure the breaking point](databasev2/01-ram-ceiling-measurement.md) | βœ… **MEASURED 2026-08-27** β€” `readiness: ready`, `status: done`; forks settled, harness landed (**148 checks**). Footprint **96.5–100 B/row** Int vs **320.6–324 B/row** text = **3.3Γ—** (not the "order of magnitude" three docs claimed), read as median-of-marginals because doublings swing a two-point slope 2Γ—. **Both predicted exits were wrong:** table storage has no checked ceiling and is **SIGKILLed** (overcommit lets `malloc` succeed, kernel kills on page touch), and swap is not latency collapse β€” 900k rows finished **148 s capped-with-swap vs 150 s uncapped**, ~1%, returning 0 while serving from disk. **Ack-after-fsync survives an OOM kill:** ~40 000 rows recovered as an intact prefix, gated as the `ceiling` leg. Also measured: **random reads over an oversized table collapse 273Γ—** (1.85M vs 6 771 reads/s, p99 1 Β΅s vs 487 Β΅s) β€” so the two access patterns sit ~270Γ— apart under the same pressure, and departure is a **step, not a curve**. Replay measured too: **β‰ˆ5.5 Β΅s/record, 1.9Γ— history penalty** (10M records β‰ˆ 55 s of boot) β€” iteration 3's missing "before", now gated. Iteration 2's budget dependency is **removed, not satisfied** β€” there is no "swap onset" to derive it from | -| 2 | [per-table storage: `durable` and `resident`](databasev2/02-table-storage-modes.md) | πŸ”„ **the language enrichment β€” the `durable` half is DONE and usable.** Two optional `@table` keys, `durable: true\|false` and `resident: all\|keys`, both defaulting to today's behaviour (all 28 existing declarations compile unchanged, no golden moved). Landed: the grammar, WO-E224 (a durable `ref` into a volatile table is refused), `.wob` v7 carrying both properties in spare `flags` bits, `durable: false` actually skipping the WAL (measured: 50 inserts β†’ 1500 bytes durable, **0** volatile) with a mode-mismatch startup refusal, plus offset capture and read-a-row-from-an-offset. Outstanding: 5c/5d (the idβ†’offset map and rewiring `wo_row_ptr`'s 11 call sites, slab scans and `@unique`/FK across the boundary β€” not yet written up), the two runtime refusals, and closeout. [spec](../superpowers/specs/2026-08-26-table-residency-design.md) Β· [plan](../superpowers/plans/2026-08-26-table-residency.md) | +| 2 | [per-table storage: `durable` and `resident`](databasev2/02-table-storage-modes.md) | βœ… **CLOSED 2026-09-10 β€” the language enrichment: grammar, the `durable` half, keys-resident CRUD, the measurement and task 6a's refusal are all landed.** Two optional `@table` keys, `durable: true\|false` and `resident: all\|keys`, both defaulting to today's behaviour (all 28 existing declarations compile unchanged, no golden moved). Landed: the grammar, WO-E224 (a durable `ref` into a volatile table is refused), `.wob` v7 carrying both properties in spare `flags` bits, `durable: false` actually skipping the WAL (measured: 50 inserts β†’ 1500 bytes durable, **0** volatile) with a mode-mismatch startup refusal. **5c/5d landed 2026-08-29** β€” the id map stores LOG OFFSETS in place of slot indices, every reader goes through `wo_row_borrow`/`wo_row_release`, scans through `wo_row_next_id`, `@unique` and FK restrict cross the boundary, and compaction re-points the map; two memory-corruption bugs (a `delete` indexing a slab with a byte offset; a tombstone replayed as corruption) found by audit and pinned. Keys-resident **updates** lifted 2026-08-30 (delta records folded on read; [11](databasev2/11-bounded-delta-chains.md) bounds the chain). **Task 7 measured and gated 2026-08-30 (`a310496`)**: **2.55Γ—** smaller resident set, **1.53Γ—** faster than swapping under a cap, **4.2Γ—** slower reads when RAM is not tight β€” and the finding that cgroup limits charge the page cache, so the honest guidance is "fit more, not go faster"; `residency.*` rows in the baseline. **Task 6a landed 2026-09-10 and CLOSED the iteration:** refuse `durable: true` (the default) with no `WO_DATA` β€” exit 2, one stderr line naming the first durable table and all three ways forward β€” with **`WO_EPHEMERAL=1`** as the exact-value whole-program escape (refused alongside `WO_DATA` or with any other value; does not bypass the `resident: keys` refusal), and the **`.wob` v8 table bit** (`WO_CLASSF_TABLE`) so the rule binds `@table` classes only β€” the first cut, keyed on the mere absence of the volatile bit, refused every class-bearing program; seven forks auto-approved, `review_pending`. **Task 6b, the resident byte budget, moved to [5](databasev2/05-bounded-tables-eviction.md) Phase A** on 2026-09-09. [spec](../superpowers/specs/2026-08-26-table-residency-design.md) Β· [plan](../superpowers/plans/2026-08-26-table-residency.md) | | 3 | [WAL checkpoint](databasev2/03-wal-checkpoint.md) *(was 32)* | βœ… **LANDED 2026-08-29 β€” the chain's last link.** Compaction rewrites the log as one record per live row and swaps it in with `rename`, so **recovery is completely unchanged** and crash safety comes from the filesystem rather than from code. **2.16Γ— space reclaimed** (1 962 358 β†’ 907 094 B), **boot 114 β†’ 64 ms**, stop-the-world pause **2 651 Β΅s** against a stated 50 ms budget. Read `.dev/reference/postgresql` for it: PG *never* compacts its WAL β€” its records are page deltas, so it needs heap files, a control file, a redo pointer and a separate process. Ours are full row images, so a compacted log IS a store, which deletes all of that. `kill -9` during compaction: 40 rounds/run, 10 clean runs, and **mutation-proven** β€” against in-place rewrite instead of `rename` the battery fails every time. Outstanding: the **`resident: keys` offset map** (compaction moves every record; the obligation is recorded at the compactor) and the O(live rows) pause, ~5.5 s at 1 GB, which is what an incremental design must be bought against | -| 4 | [io_uring group commit](databasev2/04-io-uring-commit.md) *(was 23)* | βœ… **part A LANDED 2026-08-28 β€” group commit**, one barrier per drain instead of one per statement (the engine was fsync-per-STATEMENT, not per commit; the story's premise was wrong). Shard 0 holds each reply, commits once when its queue empties, releases all β€” so a writer is acked after the barrier carrying ITS record. **β‰ˆ2.9Γ— durable write throughput, β‰ˆ2.1Γ— lower p50**, two measurement methods agreeing (2.9Γ— controlled, 3.5Γ— s1-vs-sN); mean batch 5.43, peak 57. A durability failure is now **fatal (exit 74), not a catchable `WO_T_IO`** β€” replacing three behaviours that disagreed, two of which admitted leaving RAM ahead of disk. **What it did NOT do:** `durable.sN.mixwrite` 480β†’492 (unchanged β€” that workload does 20 writes at C=4, mean batch 1.01) and `seed` unchanged (serial writers have nothing to batch with). **This row used to say "close the 66Γ— gap"; that target was mis-stated** β€” the gap is two problems and part A fixes only the concurrent one. ⬜ part B (io_uring) **needs re-brainstorming**, not starting on the old premise | -| 5 | [Bounded tables and eviction](databasev2/05-bounded-tables-eviction.md) | ⬜ a declared capacity + refuse/evict/back-pressure, and a process-level pressure signal that sheds **before** the allocator or OS gets involved β€” turning the invisible failure into a managed one | +| 4 | [io_uring group commit](databasev2/04-io-uring-commit.md) *(was 23)* | βœ… **part A LANDED 2026-08-28 β€” group commit**, one barrier per drain instead of one per statement (the engine was fsync-per-STATEMENT, not per commit; the story's premise was wrong). Shard 0 holds each reply, commits once when its queue empties, releases all β€” so a writer is acked after the barrier carrying ITS record. **β‰ˆ2.9Γ— durable write throughput, β‰ˆ2.1Γ— lower p50**, two measurement methods agreeing (2.9Γ— controlled, 3.5Γ— s1-vs-sN); mean batch 5.43, peak 57. A durability failure is now **fatal (exit 74), not a catchable `WO_T_IO`** β€” replacing three behaviours that disagreed, two of which admitted leaving RAM ahead of disk. **What it did NOT do:** `durable.sN.mixwrite` 480β†’492 (unchanged β€” that workload does 20 writes at C=4, mean batch 1.01) and `seed` unchanged (serial writers have nothing to batch with). **This row used to say "close the 66Γ— gap"; that target was mis-stated** β€” the gap is two problems and part A fixes only the concurrent one. **Part B re-brainstormed 2026-09-10** (codd-shoney): ten forks named, 1–5 and 8–10 settled under `review_pending`; forks 6 (lintor's kernel-floor answers) and 7 (cyril's tmpfs-vs-ext4 ceiling: **GO**, tmpfs `mixread.p99` 91–112 Β΅s on the RAM figure, ext4 3902–4307 Β΅s, bar for B8 ext4 `mixread.p99` ≀ 225 Β΅s / β‰₯ 15k reads/s / `s1.seed.p50` within 15% of 213 Β΅s / `wmix.ops_sec` β‰₯ 6017) are **settled in substance but fold pending** β€” both answers sit in `.dev/zack/databasev2-4b.md`, not yet folded into the story. πŸ”„ `status: in-progress`, **`readiness: refine`** until the fold lands | +| 5 | [Bounded tables and eviction](databasev2/05-bounded-tables-eviction.md) | ⬜ **brainstormed to `readiness: ready` 2026-09-10** by codd-shoney: twelve forks settled, `review_pending` β€” rows per table (`max_rows`/`on_full`), bytes per process (`WO_DB_MB`), default = cgroup limit or `MemAvailable` minus boot RSS, refuse on breach, no eviction on durable tables, `drop_oldest` volatile-only, chunk-rounded estimate, `.wob` v9. Phase A is engine-only and startable now; a prebuild-feature brief is recommended before Phase B. `status: pending` β€” not started | | 6 | [Cold tiering](databasev2/06-cold-tiering.md) | ⚠ **largely superseded by 2** β€” `resident: keys` took the ceiling-raising role; its user-space-working-set premise was rejected for the kernel page cache. Mostly forks: which shape, whether the index itself fits, whether the *language* surfaces the fault cost, and whether `@unique` on a cold table is refused outright. A paged B-tree stays rejected β€” if tiering needs one, reject tiering | -| 7 | [Single-file store](databasev2/07-single-file-db.md) *(was 33)* | ⬜ `WO_DATA=.db`; driver-only, independent | +| 7 | [Single-file store](databasev2/07-single-file-db.md) *(was 33)* | βœ… **CLOSED 2026-09-10.** `WO_DATA=.db`; driver-only, independent. `wo_wal_resolve_data_path` + the two refusals (`b31bd40`), compaction/migration temps pinned beside a file-form log (`ccee2d0`), the contract + `CODE-LOGIC.md` paragraphs (`f1985ba`), the gate leg (`e274f4a` + `aaea6b2`, codd-cyril): `just residency` **32/0**, `db-bench --quick --wo-data-file` **181 checks, 5 failures** β€” the same 5 as the plain directory form (`residency.keys.fit` rc 74, not this defect). Open item: the gate runs a stale `runtime/wovm` unless it is rebuilt first β€” a follow-up for codd-cyril | | 8 | [Query grammar from corpora](databasev2/08-query-grammar-corpus.md) *(was 27)* | ⬜ whole-query `count`, `exists`; independent | | 9 | [Cross-program tables](databasev2/09-cross-program-tables.md) *(was 20)* | ⏸ hold β€” attach to a running program's database over local IPC | | 10 | [Keypair attach auth](databasev2/10-keypair-attach-auth.md) *(was 21)* | ⏸ hold β€” program identity as a keypair; needs 9 | | 11 | [Bounded delta chains](databasev2/11-bounded-delta-chains.md) | βœ… **LANDED 2026-08-30.** A `resident: keys` row's delta chain is bounded in the UPDATE path, because the checkpoint is blind to per-row chain length β€” it thresholds on whole-log bytes, so one hot row can grow an unbounded chain inside a log that never trips compaction. The fold now reports hop count (free β€” the walk already visited every hop), and past `WO_DELTA_MAX_HOPS` (16) the update writes a full row image instead of a delta, resetting depth to 0. **Two things the tests corrected.** The flattened image is a `WO_WAL_UPDATE`, not an `INSERT`: the row's original INSERT is already in a live log, so a second one for the same id is a duplicate that replay correctly refuses as corruption β€” INSERT is right only for compaction, which builds a *fresh* log. And the **proportional ceiling was removed as dead code**: with the absolute term at 64 MiB, garbage large enough to reach a 256 MiB ceiling has already tripped it, so the branch was unreachable. Borrowing both constants from postgres was the wrong inference β€” PG needs two because it thresholds on *tuples* with its pair at opposite ends (base 50, max 1e8); this thresholds on *bytes*, where one constant does both jobs. Found by trying to write a test for the ceiling and finding no input could reach it. Four tests: depth stays bounded across 2K+2 updates, a flattened chain replays, a delta on an **indexed** column composes with flattening (checked at every step across the bound and after restart β€” found no product defect), and the policy's absolute term with its boundary. `test_wal` **5700 pass / 0 fail**; `wovm-test` and `woc-test` green. **One criterion is weaker than written:** the replay check asserts an expected value, not a `resident: all` oracle table. [spec](../superpowers/specs/2026-08-30-bounded-delta-chains-design.md) | | 12 | [Schema migrations](databasev2/12-schema-migrations.md) | βœ… **LANDED 2026-08-31.** A `@table` class is the schema, the log is the database, and boot now compares them β€” before this, an added or deleted field turned a healthy `WO_DATA` into "corruption" and reordering declarations silently decoded rows into the wrong class. Landed: `WO_WAL_SCHEMA` head record (written LAZILY ahead of the first real record β€” an eager head broke `durable: false`'s documented zero-bytes contract by 75 bytes and the gate caught it), a name-keyed diff whose refusals are per-class POISONS that bite only when a record of the class is met, and a record-level TRANSCODE: cids remap by name including inside stored owned values, deleted values freed, added fields zero-filled, delta back-pointers rewritten through an offset map with deltas on deleted fields SPLICED out; temp+fsync+rename, compaction's crash discipline. **Two bugs the tests forced out:** a poisoned class skipped plan identity so the retype refusal fell through to generic "corruption" (the message this iteration exists to replace), and early `goto corrupt` freed uninitialized memory. End-to-end: `migrating \`Note\`: +flag` then `flag=0`; retype refuses naming `val`, exit 2, old binary still boots the refused log. 21 new tests, `test_wal` **5966/0**; wovm/woc/site/residency gates green. v2 holds rename (`@renamed_from`), retypes, and data/seed migrations. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) | +| 13 | [Fresh-log keys-resident seed SEGV](databasev2/13-fresh-log-keys-seed-segv.md) | βœ… **FIXED 2026-09-10.** Defect found while smoke-testing 7 (independent of it): `seed` of `docs/examples/residency` on a fresh log SEGV'd rc 139 in both `WO_DATA` forms β€” `wo_wal_fold_row_at` wrote an unguarded `*msg` while `wo_idx_probe` borrows with `msg = NULL`, entering the "record header is malformed" arm; root cause confirmed: the databasev2 12 schema head was staged lazily after the first keys-resident row's offset was captured (`db.c`'s `koff`). Fixed: `wo_wal_next_offset` now stages the head first (`6310078`) + the fold tolerates a NULL `msg` (`1b6750d`); `test_wal` 6629β†’6660, `make -C runtime test` 21 suites **8462/0**, `just residency` **32/0**. Also found and fixed: `scripts/residency-accept.sh:155` never checked `seed`'s rc, so the 20/0 residency gate was green while `seed` crashed under it (`e274f4a`, databasev2 7 task 4). `residency.keys.fit` rc 74 confirmed a **separate**, still-open bug (different code path) | --- @@ -1579,8 +1583,9 @@ proven before the runtime and `Resp` are touched. --- -⏸ **Held** (2026-08-21, developer decision): 18, 20, 21, 25, 26, 27, 28, -29 β€” every story carrying `status: hold` in its frontmatter (25's story +⏸ **Held** (2026-08-21, developer decision): 20, 21, 25, 26, 27, 28, +29 β€” every story carrying `status: hold` in its frontmatter (18's hold lifted +2026-09-11, above; 25's story file removed; its [plan doc](../superpowers/plans/2026-08-01-http-service-layer.md) remains). Half-done branches (ipc-attach, keypair-auth) keep their diff --git a/docs/stories/databasev2/00-story.md b/docs/stories/databasev2/00-story.md index 65eca09..8e235ab 100644 --- a/docs/stories/databasev2/00-story.md +++ b/docs/stories/databasev2/00-story.md @@ -150,25 +150,32 @@ before its mechanism existed; the history is in | # | Iteration | Delivers | Needs | | --- | --- | --- | --- | -| 1 | [RAM ceiling: measure the breaking point](01-ram-ceiling-measurement.md) | πŸ”„ **measured 2026-08-27**: footprint per shape (3.3Γ— apart), the two silent exits (SIGKILL vs swap-serving-from-disk at ~uncapped speed), and ack-after-fsync surviving an OOM kill. Also measured: the **273Γ— random-read collapse** over an oversized table, and replay at **β‰ˆ5.5 Β΅s/record with a 1.9Γ— history penalty** β€” iteration 3's "before" | nothing; extends iteration 22's harness | -| 2 | [per-table storage](02-table-storage-modes.md) | the grammar: `durable: true\|false` and `resident: all\|keys`, per table, replacing the global `WO_DATA` all-or-nothing. **In progress β€” the `durable` half is done** | 1 for the budget default | +| 1 | [RAM ceiling: measure the breaking point](01-ram-ceiling-measurement.md) | βœ… **measured 2026-08-27**: footprint per shape (3.3Γ— apart), the two silent exits (SIGKILL vs swap-serving-from-disk at ~uncapped speed), and ack-after-fsync surviving an OOM kill. Also measured: the **273Γ— random-read collapse** over an oversized table, and replay at **β‰ˆ5.5 Β΅s/record with a 1.9Γ— history penalty** β€” iteration 3's "before" | nothing; extends iteration 22's harness | +| 2 | [per-table storage](02-table-storage-modes.md) | βœ… **closed 2026-09-10.** the grammar: `durable: true\|false` and `resident: all\|keys`, per table, replacing the global `WO_DATA` all-or-nothing. Grammar, the `durable` half, keys-resident CRUD and task 7's measurement landed by 2026-08-30; task 6a β€” refuse `durable: true` without `WO_DATA`, `WO_EPHEMERAL=1` as the whole-program escape, the `.wob` v8 table bit so the rule binds `@table` classes only β€” landed 2026-09-10; the byte budget (6b) moved to 5 | 3 (the offset map survives compaction β€” landed); no longer 1, since the budget moved to 5 on 2026-09-09 | | 3 | [WAL checkpoint](03-wal-checkpoint.md) *(was language 32)* | snapshot + truncate: disk reclaimed, replay bounded | 4 composes | -| 4 | [io_uring group commit](04-io-uring-commit.md) *(was language 23)* | close the 66Γ— durable/RAM write gap (4.5k vs 297k inserts/s) | the arc (landed) | -| 5 | [Bounded tables and eviction](05-bounded-tables-eviction.md) | a capacity a `ram` table may not exceed, and what happens when it does | 2 | +| 4 | [io_uring group commit](04-io-uring-commit.md) *(was language 23)* | one barrier per DB-actor drain instead of one per statement β€” part A landed 2026-08-28 (β‰ˆ2.9Γ— concurrent durable writes); the "66Γ— gap" this row used to name is a serial writer's latency, which part B must re-brainstorm. **Part B re-brainstormed 2026-09-10**: forks 1–5/8–10 settled (`review_pending`); forks 6/7 settled in substance (GO β€” tmpfs `mixread.p99` on the RAM figure, ext4 3902–4307 Β΅s) but **fold pending** β€” see `.dev/zack/databasev2-4b.md`. `status: in-progress`, `readiness: refine` until folded | the arc (landed) | +| 5 | [Bounded tables and eviction](05-bounded-tables-eviction.md) | a capacity a `ram` table may not exceed, and what happens when it does; **since 2026-09-09 also the resident byte budget** (iteration 2's former task 6b) as its Phase A. **Brainstormed to `readiness: ready` 2026-09-10** (codd-shoney): twelve forks settled, `review_pending`; Phase A is engine-only and startable, a prebuild brief is recommended before Phase B. `status: pending` β€” not started | 1 (the budget default follows a measurement) and 2 (the mode a bound attaches to) | | 6 | [Cold tiering](06-cold-tiering.md) | ⚠ **largely superseded by 2** β€” `resident: keys` is the ceiling-raiser. Its premise (a user-space resident working set) was rejected in favour of the kernel page cache. Revisit only with a measurement showing the page cache insufficient | β€” | -| 7 | [Single-file store](07-single-file-db.md) *(was language 33)* | `WO_DATA=.db` β€” a file path IS the store | independent | -| 8 | [Query grammar from corpora](08-query-grammar-corpus.md) *(was language 27)* | whole-query `count`, `exists` | independent | +| 7 | [Single-file store](07-single-file-db.md) *(was language 33)* | βœ… **closed 2026-09-10.** `WO_DATA=.db` β€” a file path IS the store: directory or trailing `/` stays byte-identical to today; otherwise the path IS the log, created if absent behind an existing parent, refused (exit 2, naming path + parent) on a missing parent or a non-regular/non-directory path. Compaction and migration temps land beside the file-form log, pinned by a test + a mutation control. Gate leg (task 4, codd-cyril): `just residency` **32 checks, 0 failures**; `db-bench --quick --wo-data-file` **181 checks, 5 failures**, the same 5 as the directory form (`residency.keys.fit` rc 74, databasev2 13's sibling bug, not this defect) | independent | +| 8 | [Query grammar from corpora](08-query-grammar-corpus.md) *(was language 27)* | whole-query `count` (landed 2026-08-16 from the skill-catalog corpus); `exists` waits for a corpus that forces it | independent | | 9 | [Cross-program tables](09-cross-program-tables.md) *(was language 20)* | attach to a running program's database over local IPC | independent | | 10 | [Keypair attach auth](10-keypair-attach-auth.md) *(was language 21)* | program identity as a keypair; mutual challenge–response | 9 | -| 11 | [Bounded delta chains](11-bounded-delta-chains.md) | cap a keys-resident row's delta chain in the update path, and give the compaction policy an absolute term + ceiling | 2 (fixes a limitation it shipped) | +| 11 | [Bounded delta chains](11-bounded-delta-chains.md) | cap a keys-resident row's delta chain in the update path, and give the compaction policy an absolute garbage term (`WO_CKPT_ABS_BYTES`; a separate ceiling was tried and removed) | 2 (fixes a limitation it shipped) | +| 12 | [Schema migrations](12-schema-migrations.md) | βœ… **landed 2026-08-31**: the log describes itself (`WO_WAL_SCHEMA` head record, kind 5); boot diffs by name, transcodes add/delete record by record, refuses everything else by name | 2 (the v7 descriptor, and the delta record it rewrites) | +| 13 | [Fresh-log keys-resident seed SEGV](13-fresh-log-keys-seed-segv.md) | βœ… **fixed 2026-09-10.** A `resident: keys` table's first insert on a fresh log SEGV'd (`wo_wal_fold_row_at` wrote an unguarded `*msg`; the schema head was staged after the first row's offset was captured). Fixed: `wo_wal_next_offset` stages the pending head before returning an offset (`6310078`), plus a NULL-`msg` guard in the fold (`1b6750d`); `test_wal` 6660/0, `make -C runtime test` 21 suites 8462/0, `just residency` 32/0. **Not** the same defect as `residency.keys.fit` rc 74 (compaction/replay of keys-resident offsets), which stays open under codd.md's "Next bugs" | 2 (the offset map), 12 (the schema head record) | +| 14 | [The shop workload](14-shop-workload.md) | what an order-taking web app needs from the store: ordered index + range probe, `skip`, composite unique + check rules, on-delete policy, export/import; group-by carried as a criterion (language track) | 2 (keys-resident index shape), 9 (attach), language 18 (implicit block for cascade) | ``` An arrow points AT the iteration that NEEDS the other. -1 ──▢ 2 ◀── 3 2 needs 1 (budget from a measurement) and 3 (the - β”‚ offset map survives compaction). Since 5d, 3 also - β–Ό calls 2's row API β€” the coupling runs both ways. - 5 5 needs 2. Nothing needs 5. + 2 ◀── 3 2 needs 3 (the offset map survives compaction). Since + β”‚ 5d, 3 also calls 2's row API β€” the coupling runs both + β–Ό ways. 2 no longer needs 1: its budget moved to 5 +1 ──▢ 5 (2026-09-09). 5 needs 1 (the budget default follows a + measurement) and 2 (the mode a bound attaches to). + Nothing needs 5. +2 ──▢ 11, 12 both need 2: 11 bounds a chain 2 shipped, 12 transcodes + the descriptor and the records 2 defined. 4 composes with 3 on the WAL commit path; NEITHER needs the other. Executed 4 then 3 (chain 5, then 6). @@ -177,7 +184,8 @@ An arrow points AT the iteration that NEEDS the other. 9 ──▢ 10 ``` -Order rationale: **1 before 2** because the budget default should follow from a +Order rationale: **1 before 5** β€” it read "1 before 2" until 2026-09-09, when +the budget moved to 5 β€” because the budget default should follow from a measurement, not a guess. **3 and 4 matter to 2** for the same reason tiering onto a never-truncating log would have: `resident: keys` rebuilds its offset map by scanning the whole log at boot until 3's snapshot persists it. diff --git a/docs/stories/databasev2/01-ram-ceiling-measurement.md b/docs/stories/databasev2/01-ram-ceiling-measurement.md index 3abfc9a..103c4c0 100644 --- a/docs/stories/databasev2/01-ram-ceiling-measurement.md +++ b/docs/stories/databasev2/01-ram-ceiling-measurement.md @@ -243,7 +243,11 @@ Outstanding: **Iteration 2 must pick its budget on other grounds** (host RAM fraction, or an explicit developer-declared figure) rather than waiting on a number this iteration cannot produce. This is the most important thing this slice learned - and it removes a dependency rather than satisfying it. + and it removes a dependency rather than satisfying it. **Redirected + 2026-09-09:** the budget itself moved from iteration 2 to + [5](05-bounded-tables-eviction.md) Phase A, so this finding β€” pick the + default on other grounds β€” is 5's input now, and iteration 2 no longer needs + this iteration at all. - **Given** rising fractions of the cap, **when** latency is sampled, **then** the p99 departure point is recorded. Partially, and now with a real answer elsewhere: `p99_departure_decile` stays 0 because the footprint legs never @@ -293,9 +297,10 @@ Outstanding: runs that differ only in their cap belongs to the engine. Both legs read the same Weyl key order (`i*2654435761 mod n` β€” no RNG in the language, and none needed) so residency is the only variable. -7. **The ceiling leg asserts `rc`, never records it.** When iteration 2's byte - budget lands, death should become a checked refusal β€” the gate must not fail - on that improvement. +7. **The ceiling leg asserts `rc`, never records it.** When the byte budget + lands (iteration 2's until 2026-09-09; now + [5](05-bounded-tables-eviction.md) Phase A), death should become a checked + refusal β€” the gate must not fail on that improvement. ## History β€” four corrections worth keeping diff --git a/docs/stories/databasev2/06-cold-tiering.md b/docs/stories/databasev2/06-cold-tiering.md index 1047ccc..46ee28b 100644 --- a/docs/stories/databasev2/06-cold-tiering.md +++ b/docs/stories/databasev2/06-cold-tiering.md @@ -12,6 +12,12 @@ readiness: refine > [3](03-wal-checkpoint.md) so the log this builds on does not grow forever, > and [5](05-bounded-tables-eviction.md) for the policy machinery. > +> **Correction, 2026-09-10:** the Needs line above is superseded prose, kept +> for the iteration's original reasoning, not a live dependency β€” the +> story's own sequence table lists this iteration's Needs as "β€”" +> ([00-story.md:158](00-story.md)) and its ASCII graph says "superseded by +> 2 β€” not sequenced" ([00-story.md:180](00-story.md)). +> > **⚠ LARGELY SUPERSEDED 2026-08-27 by [iteration 2](02-table-storage-modes.md).** > This iteration was written to implement a `cold` mode. That mode no longer > exists: the brainstorm replaced it with `resident: all | keys`, and @@ -28,7 +34,7 @@ readiness: refine > the reason the engine avoids `O_DIRECT`. > > **What may still be left:** if measurement after 5c/5d (landed 2026-08-29, -> still unmeasured β€” that is iteration 2's task 7) shows the page cache +> measured by iteration 2's task 7 on 2026-08-30 β€” see below) shows the page cache > insufficient for some workload, a user-space working set becomes arguable > again β€” but only with that number in hand, which is the opposite of how this > file was written. @@ -40,8 +46,11 @@ readiness: refine > the number `resident: keys` must **beat**, since it `pread`s through the page > cache, which gets readahead and a shared cache. So this file revives if and > only if 5c/5d measures the page-cache path landing near 273Γ— rather than well -> below it. Until that measurement exists, neither outcome is assumed. Until then treat the design questions below as answered -> elsewhere and the phases as void. Its genuinely durable contribution is its +> below it. **Measured 2026-08-30 (iteration 2, task 7):** under the same memory +> cap `resident: all` collapsed **105Γ—** from its uncapped throughput and +> `resident: keys` only **16Γ—**, 1.53Γ— faster than swapping β€” well below 273Γ—, +> so by this file's own criterion it stays superseded. Treat the design +> questions below as answered elsewhere and the phases as void. Its genuinely durable contribution is its > fork list, especially "does the language surface the fault cost at the *use* > site" β€” still open, and still the largest question about what writeonce is. > diff --git a/docs/stories/databasev2/09-cross-program-tables.md b/docs/stories/databasev2/09-cross-program-tables.md index d485f3b..bf7c21e 100644 --- a/docs/stories/databasev2/09-cross-program-tables.md +++ b/docs/stories/databasev2/09-cross-program-tables.md @@ -78,7 +78,11 @@ readiness: refine - **Given** the employee sample running as A with its departments and employees tables, - **when** a second sample program (a thin reporting client) attaches - read-only and runs the GroupBy report over `a.Employee`, + read-only and runs the sample's department report over `a.Employee` + (as written in `docs/examples/employee-list/main.wo` it is a + `group … by … into` query, which the compiler still refuses β€” + `types.ml` "group-by aggregation is not supported yet" β€” so group-by + aggregation lands first or the report is rephrased), - **then** it prints the same report the owner prints β€” the demonstration that attach + query compose. @@ -133,6 +137,15 @@ A's checkout); A **exports a schema file** (a `.wob`-adjacent digest of its class table) that B's manifest points at (decoupled, but a new artifact with a staleness story); or shared type definitions in a common module both import (cleanest language story, needs the module system to span projects). +**Annotation, 2026-09-10:** that module system is language-track work, not +this iteration's. The closest existing story is [language-runtime-database +15](../language-runtime-database/15-deps-package-manager.md) (`wo.toml +[deps]`, git fetch β€” done, 2026-08-18), which lets `use ` resolve +into a fetched dependency's module tree; two separately-run programs naming +the same dependency already share its types at build time, but no story +addresses schema-sharing between two independently *running* programs +specifically β€” no dedicated language story yet for that; this option +remains a fork for the language track. A runtime schema handshake must exist regardless β€” B's compiled expectation of `a.Employee`'s shape is verified against A's live class table at attach, and a mismatch refuses the attachment with both sides' shapes named. diff --git a/docs/stories/databasev2/10-keypair-attach-auth.md b/docs/stories/databasev2/10-keypair-attach-auth.md index 3c8aaa8..5df76ff 100644 --- a/docs/stories/databasev2/10-keypair-attach-auth.md +++ b/docs/stories/databasev2/10-keypair-attach-auth.md @@ -114,6 +114,13 @@ component with its provenance pinned in the tree β€” the doctrine's spirit is "no dependency sprawl", not "write your own constant-time field arithmetic". +**Annotation, 2026-09-10:** an in-tree answer now exists: runtime-v2 9 +landed constant-time RSA-PSS and ECDSA-P256 signing plus X.509 parsing +(`runtime/src/crypto.c`, `tls.c`), runtime-v2 8 the AEADs; libsodium would +violate the zero-dependency doctrine ([docs/00-principles.md principle +2](../../00-principles.md)). Not a decision β€” Ed25519-vs-reuse is still this +fork's open question. + **2. Key generation and storage.** Options: a `woc keygen` subcommand (keys are a toolchain concern), or first-boot generation by the runtime into the data directory (keys are a runtime concern, zero setup). Leaning: diff --git a/docs/stories/porch/00-story.md b/docs/stories/porch/00-story.md index 3ef6473..70f8973 100644 --- a/docs/stories/porch/00-story.md +++ b/docs/stories/porch/00-story.md @@ -49,7 +49,7 @@ risky work starts. | # | Iteration | Delivers | Needs | | --- | --- | --- | --- | -| 1 | [Store-backed middleware](01-store-backed-middleware.md) | rate limiting + idempotency over a `@table` store, serialized through a sharded actor pool | πŸ”„ in progress; needs no new primitive (`call`/`send`/`monitor`/`time.after` all landed) | +| 1 | [Store-backed middleware](01-store-backed-middleware.md) | rate limiting over a sharded actor pool β€” exact counting, restart-durable | βœ… **done** 2026-08-30; re-scoped to the limiter alone | | 2 | [Randomness and cookies](02-randomness-and-cookies.md) | a `random_bytes` runtime builtin, repeated response headers, `Cookie:` parsing, signed cookies | a language-track builtin (phase A) | | 3 | [Sessions](03-sessions.md) | server-side sessions, idle + absolute timeout, revocation | 2 | | 4 | [CSRF](04-csrf.md) | token mint/verify, trusted origins, single-use tokens | 2, 3 | @@ -57,6 +57,8 @@ risky work starts. | 6 | [Streaming core](06-streaming-core.md) | incremental response writes and chunked framing β€” the seam three iterations wait on | nothing new, but it changes `Resp` | | 7 | [SSE and compression](07-sse-and-compression.md) | server-sent events, gzip/deflate | 6 | | 8 | [Static files and lifecycle](08-static-and-lifecycle.md) | byte ranges, cache headers, directory listing, lifecycle hooks, the small middleware everyone ships | 6 | +| 9 | [Idempotent replay](09-idempotent-replay.md) | idempotent replay of unsafe requests, the actor running the route handler | ⏸ built and reverted; blocked on [language 41](../language-runtime-database/41-actor-arena-crash.md) | +| 10 | [Memory features over `@table`](10-memory-features-over-table.md) | TTL cache, `@table` feature flags, durable job queue β€” the three `.wo` pieces split out of language 18 on 2026-09-11 | stub, `readiness: refine`; needs [language 18](../language-runtime-database/18-memory-db-features.md)'s `transaction { }` for the jobs demo, and 1–3 | ``` 1 ─ independent, start here @@ -71,7 +73,7 @@ risky work starts. | Not porch's | Owner | | --- | --- | | typed binding of query/params/form into a class | language: [`@derive`](../language-runtime-database/29-compile-time-metaprogramming.md) β€” reflection is forbidden by principle 13 | -| TTL cache, `transaction { }`, durable job queue | language: [iteration 18](../language-runtime-database/18-memory-db-features.md) | +| `transaction { }` | language: [iteration 18](../language-runtime-database/18-memory-db-features.md) β€” the engine + language half; TTL cache, `@table` feature flags and the durable job queue **moved into this track** as [iteration 10](10-memory-features-over-table.md) on 2026-09-11 | | a `proxy` middleware | language: [iteration 38](../language-runtime-database/38-content-platform-capabilities.md) β€” needs `net.connect`, βœ… landed 2026-09-07 (id 110; plus `net.connect_tls` for an HTTPS upstream, runtime-v2 9). Buildable now | | metrics, profiling, per-change CI, fuzzing | [runtime-v2 7](../runtime-v2/07-observability.md) β€” observability (was language iteration 30; metrics/profiling/trace-on-trap; CI + fuzz are tooling, split out) | | TLS | βœ… [runtime-v2 9](../runtime-v2/09-in-process-tls.md) β€” in-process TLS 1.3 both directions (2026-09-09); porch can terminate inbound TLS with `net.accept_tls`, no front proxy required. The proxy-termination doctrine is retired |