From 296efb52ed11c5d893bde5278d7331e31e710215 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Tue, 15 Sep 2026 01:03:49 +0200 Subject: [PATCH] =?UTF-8?q?docs(db2-ephemeral):=20databasev2=202=20closes?= =?UTF-8?q?=20=E2=80=94=20task=206a=20contract,=20forks=201=E2=80=937,=20t?= =?UTF-8?q?he=20README=20sweep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - story 02: `status: done`, `review_pending` (forks 1–7 auto-approved for autonomy); progress rows 6a ✅, 6b ➡ databasev2 5 Phase A, 7 `a310496`; 5c/5d rows cite the `dev` hashes (the pre-merge ones were unreachable); task 6a's Given/When/Then met; Info records the seven forks (sentinel over `:memory:`, its rules, the refusal contract, startup-only, the budget leaves for 5, library-owned tables bind consumers, the v8 table bit); History keeps the first cut that refused every class-bearing program - database/src/CODE-LOGIC.md: "Startup refusal + WO_EPHEMERAL" — contract, hatch, table bit, measured blast radius, deferred items, proof; the dispatcher paragraph no longer says a failed commit un-applies the row (fatal since databasev2 4 part A; WO_T_IO unreachable from a write path) - residency spec + plan: task 6 items annotated with the 2026-09-09 decisions; the byte budget marked moved to databasev2 5 - README, seven example READMEs and four guides carry the one-line rule (durable default refuses without WO_DATA; WO_EPHEMERAL=1; durable: false); shop's RAM-only command sets the sentinel Co-Authored-By: Claude Fable 5.1 (cherry picked from commit 2c3531998124042fe736388e8b926abda3841194) --- README.md | 2 + database/src/CODE-LOGIC.md | 76 +++++++- docs/examples/db-actor/README.md | 2 +- docs/examples/db-bench/README.md | 3 +- docs/examples/employee-list/README.md | 5 + docs/examples/residency/README.md | 2 +- docs/examples/shop/README.md | 4 +- docs/examples/skill-catalog/README.md | 2 +- docs/examples/web-app/README.md | 2 + docs/guides/deploying-site.md | 2 +- docs/guides/log-structured-rows.md | 3 +- docs/guides/releasing.md | 2 +- docs/guides/updating-site.md | 2 +- .../databasev2/02-table-storage-modes.md | 176 ++++++++++++++---- .../plans/2026-08-26-table-residency.md | 21 ++- .../2026-08-26-table-residency-design.md | 23 ++- 16 files changed, 266 insertions(+), 61 deletions(-) diff --git a/README.md b/README.md index 6abc206..28bc4a1 100644 --- a/README.md +++ b/README.md @@ -302,6 +302,8 @@ WO_DATA=./data ./target/myproject seed WO_DATA=./data ./target/myproject report # a fresh process still sees the data ``` +A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. + --- ## Worked examples diff --git a/database/src/CODE-LOGIC.md b/database/src/CODE-LOGIC.md index 03b528b..ac703df 100644 --- a/database/src/CODE-LOGIC.md +++ b/database/src/CODE-LOGIC.md @@ -84,11 +84,15 @@ succeed and the file is the only artifact beside a decoy sibling directory). One dispatcher, the builtin contract (0 ok, else WO_T_* + msg). The engine handles ride `wo_rt.db` / `wo_rt.wal` as opaque pointers set by main.c — -NULL db traps WO_T_DB, NULL wal means RAM-only (the corpus's mode; WO_DATA -opts into durability). Insert's contract: RAM apply through the row API, -then stage + commit BEFORE returning — the builtin's return is the -acknowledgment, so a failed commit un-applies the row and traps WO_T_IO -rather than acknowledging what disk never got. +NULL db traps WO_T_DB, NULL wal means RAM-only — reachable only under +WO_EPHEMERAL=1 (or with no durable `@table` in the module) since databasev2 2 +task 6a: a program with any durable `@table` (the default) refuses to start +without WO_DATA; WO_EPHEMERAL=1 opts into a RAM-only run (the corpus's mode), +@table(durable: false) opts a table out. Insert's contract: RAM apply through +the row API, then stage + commit BEFORE returning — the builtin's return is +the acknowledgment. Once RAM has mutated the outcomes are durable or process +death (`wo_wal_commit_fatal`, `wo_wal_stage_fatal`): a failed commit is +fatal, there is no un-apply, and WO_T_IO is unreachable from a write path. ## Verifying a change @@ -400,3 +404,65 @@ A `@table` class is the schema; the log is the database; boot compares them. - **Legacy logs** (no head record) replay exactly as before and adopt the head at their next compaction. v1 verbs are add and delete only; rename wants `@renamed_from` (v2), data/seed migrations are v2. + +## Startup refusal + WO_EPHEMERAL (databasev2 2 task 6a, 2026-09-09/10) + +`main.c`, startup only. The engine, `db.c` and `wal.c` are untouched. + +- **Contract.** With `WO_DATA` unset or empty and no `WO_EPHEMERAL`, the first + class whose flags carry `WO_CLASSF_TABLE` and lack `WO_CLASSF_VOLATILE` + (a `@table` with `durable: true`, the default) is a startup refusal: exit 2, + ONE stderr line naming the class and all three ways forward literally + (`WO_DATA=`, `WO_EPHEMERAL=1`, `@table(durable: false)`). The loop sits + inside the existing `!data_dir` block AFTER the `resident: keys` loop — the + keys refusal wins, and `WO_EPHEMERAL` does not rescue it (a keys table has + nowhere to read from). Both loops skip classes without the table bit. +- **Escape hatch.** `WO_EPHEMERAL` with the exact value `1`, honoured only + while `WO_DATA` is unset/empty: one boot notice line on stderr, rc 0, and the + RAM path is byte-for-byte the old one — `db.c`'s `w && table_is_durable` + guards are the only gate, no new flag in `wo_db`. Set alongside `WO_DATA` + (any value) → exit 2 `WO_EPHEMERAL=1 is incompatible with WO_DATA` — that + check runs regardless of tables. Any value but `1` → exit 2 naming the + accepted value. A module with no durable `@table` consults `WO_EPHEMERAL` + for nothing else: no notice, no value check, rc 0 as before. +- **The table bit (`.wob` v8, 2026-09-10).** The first cut keyed the refusal + on `!VOLATILE` alone, and the v7 image carried no "is a `@table`" bit: plain + classes, variant classes and the predeclared records (`Error`, `Stat`, …) + all looked durable, so EVERY class-bearing program refused without + `WO_DATA` — fibers (`Tick`), subprocess (`ConnMsg`), log-watcher + (`CronEntry`), chat. Wrong by construction: `durable:` is a `@table` + property. Fixed by `WO_CLASSF_TABLE` 0x08 (`wob.h`, `WO_CLASSF_ALL` 0x0f, + `WOB_VERSION` 8; `emit.ml` sets it from `cr_is_table`); the loader refuses + `VOLATILE`/`RESIDENT_KEYS` without it ("storage flags on a class that is + not a @table", `test_loader`), and a v7 image is refused by the version + check exactly as v7 refused v6. Blast radius after the fix, measured gate + by gate (each run without the export first; kept only where it refused): + only programs that DECLARE a durable table opt in — `oop-e2e.sh` (corpus + fixtures declare tables); `db-bench.py`'s ram/msgrate/growth/randread legs + (db-bench's tables); db-actor per-run (`notes` is default-durable; per-run + because its restart pair sets `WO_DATA` and the two are incompatible), + whose single-shard byte-exact compare drops the one notice line + (`grep -v '^wovm: WO_EPHEMERAL=1'`); chat, whose program declares no table + itself but `use`s porch, and porch's store middleware declares + `RateLimitCounter` default-durable — fork 6, a library-owned table binds + the consumer; and wmux, whose CLIENT legs (ls/new/attach/kill) run the + same default-durable image with no `WO_DATA` — the gate exports the + sentinel, every server start and the `WO_DATA`-carrying `r11cli` drop it + with `env -u`, and `client()` filters the notice because its answers are + compared byte-exactly (a wmux-track consequence worth its own look: a CLI + client of a durable server now needs the sentinel or a `WO_DATA`). fibers + (`Tick`), subprocess (`ConnMsg`) and log-watcher (`CronEntry`) need + nothing — their exports were reverted and their byte-exact compares are + as they were. Goldens: none moved — the bytecode dump prints flags by name + and no `bc/` golden declares a table; the header version is not printed. +- **Deferred, each its own later commit:** an assert in `db.c` that + `durable && !w` is unreachable outside `WO_EPHEMERAL`; an ENOENT hint when + the `WO_DATA` directory is missing (the FILE form already refuses with a + named parent since databasev2 7; the directory form still fails at the WAL + open, on purpose — byte-identical to before); SIGKILL / + rc 137 classification in the gates; `wal.c` fallocate/dir-fsync logging. +- **Proof:** `scripts/residency-accept.sh` section 7 — refusal text, RAM + round-trip under the hatch, the `WO_DATA` conflict, keys still refusing + under the hatch, a non-`1` value, and (vi) a plain class without `@table` + running with no `WO_DATA` and nothing on stderr (the corpus `methods` + fixture); `runtime/test/test_loader.c` `test_storage_flags_need_table`. diff --git a/docs/examples/db-actor/README.md b/docs/examples/db-actor/README.md index e688d27..1abb893 100644 --- a/docs/examples/db-actor/README.md +++ b/docs/examples/db-actor/README.md @@ -43,7 +43,7 @@ WO_SHARDS=1 ./docs/examples/db-actor/target/db-actor # force the local path | multi-shard, three rounds | The writer lines are asserted as a **set**, not a sequence — scheduling decides their order, and pinning it would be testing the scheduler, not the RPC. The `main` line is exact. | | both `WO_IO` backends forced | The reply park has to be plane-independent: io_uring and epoll must give the same answer, or the parking is leaking into semantics. | | single shard, byte-exact | The local path is untouched by stage 3. Any drift here means the RPC changed the non-RPC case. | -| `WO_DATA` restart pair | A worker's insert must commit on the **owner's** WAL before its ack, so a restart replays it: 2 rows, then 2+2 after a second run. This is the durability claim the RPC could most easily break. | +| `WO_DATA` restart pair | A worker's insert must commit on the **owner's** WAL before its ack, so a restart replays it: 2 rows, then 2+2 after a second run. This is the durability claim the RPC could most easily break. A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. | Run under `wovm_asan` and `wovm_tsan` as well — cross-shard message passing is exactly where a data race would hide, and TSan covering this demo is the one diff --git a/docs/examples/db-bench/README.md b/docs/examples/db-bench/README.md index bbe7d4e..a5c35eb 100644 --- a/docs/examples/db-bench/README.md +++ b/docs/examples/db-bench/README.md @@ -33,7 +33,8 @@ strictly better. Recorded as a plan deviation.) | var | effect | | --- | --- | -| `WO_DATA=` | durability on: replay `/shard-0.wal` at boot, log every write. Without it the store is RAM-only | +| `WO_DATA=` (or `WO_DATA=.db`, below) | durability on: replay `/shard-0.wal` at boot, log every write. A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. | +| `WO_EPHEMERAL=1` | **databasev2 2 task 6a:** the RAM-only opt-in the driver sets on its ram/msgrate/growth/randread legs. A `WO_DATA` exported in your shell no longer silently turns those legs durable — the two are incompatible and the run refuses loudly | | `WO_SHARDS=` | shard count. **`1` means every statement runs inline on shard 0 and group commit cannot engage** — batches form only where writes queue from other shards | | `WO_CHECKPOINT_BYTES` / `WO_CHECKPOINT_RATIO` | **databasev2 3:** the checkpoint trigger — the log must exceed the floor AND exceed the ratio times the last compaction's own size. A tiny floor forces compaction in a few writes, which is how the gate tests the policy at all; an enormous one disables it, which is how the checkpoint leg measures the same workload with and without | | `WO_WAL_STATS=1` | **databasev2 4:** print one line at exit — `walstats batches=… records=… peak_batch=… peak_staged=… compactions=… compact_us_max=… compact_us_total=… compacted_bytes=…`. Opt-in so it does not pollute every durable program's output. Mean batch is `records/batches`; **mean 1.0 means group commit is not engaging**, which is expected for a serial writer or `WO_SHARDS=1` and a bug anywhere else | diff --git a/docs/examples/employee-list/README.md b/docs/examples/employee-list/README.md index 1bdd3e8..86f66fd 100644 --- a/docs/examples/employee-list/README.md +++ b/docs/examples/employee-list/README.md @@ -30,6 +30,11 @@ and pasted — the `PASTE-…-HERE` placeholders mark exactly where. The connect-section name is the code's namespace: `[connect.employee]` is why the source says `employee.Employee`. +B declares the shapes but stores nothing, so it runs under `WO_EPHEMERAL=1`: a +program with any durable table (the default) refuses to start without +`WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` +opts a table out. + | Mode | What it proves | | --- | --- | | `employee-list list` | typed reads over the wire, `e.dept.name` ref navigation executing inside A | diff --git a/docs/examples/residency/README.md b/docs/examples/residency/README.md index d5fd692..ecce5aa 100644 --- a/docs/examples/residency/README.md +++ b/docs/examples/residency/README.md @@ -8,7 +8,7 @@ Before it, `WO_DATA` was the only switch. Set, and every table is WAL-logged; unset, and none are. Applications are not uniform — a session table is disposable, a product's stock level is not, and a catalogue large enough to matter does not fit in RAM at all. One global switch forces "everything is precious" or "nothing -is", and you pay for whichever is wrong. +is", and you pay for whichever is wrong. Since task 6a the default is enforced: a program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. ## Run it diff --git a/docs/examples/shop/README.md b/docs/examples/shop/README.md index e3bd455..b1f3001 100644 --- a/docs/examples/shop/README.md +++ b/docs/examples/shop/README.md @@ -10,9 +10,11 @@ has one concern, and the module system (one directory = one module, ``` cd docs/examples/shop woc . && WO_DATA=./data ./target/shop 8080 # durable store -./target/shop 8080 # RAM-only (dev) +WO_EPHEMERAL=1 ./target/shop 8080 # RAM-only (dev) ``` +A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. + Browse http://127.0.0.1:8080/ — products → product page → buy (stock checked and decremented) → confirmation → /orders. With `WO_DATA`, kill it and restart: the orders are still there (WAL replay). diff --git a/docs/examples/skill-catalog/README.md b/docs/examples/skill-catalog/README.md index c335d3f..3fdf6a0 100644 --- a/docs/examples/skill-catalog/README.md +++ b/docs/examples/skill-catalog/README.md @@ -27,4 +27,4 @@ subquery construct is needed; the general `exists`/`not exists` is deferred until a corpus uses a correlation a backlink cannot express. `skill-catalog seed | list | roots | count | get `, WAL-durable under -`WO_DATA`. Acceptance: `scripts/skill-catalog-accept.sh`. +`WO_DATA`. A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. Acceptance: `scripts/skill-catalog-accept.sh`. diff --git a/docs/examples/web-app/README.md b/docs/examples/web-app/README.md index b40d96a..c42c6e6 100644 --- a/docs/examples/web-app/README.md +++ b/docs/examples/web-app/README.md @@ -24,6 +24,8 @@ the token comes from the `WA_TOKEN` env var. woc . # fetches deps, builds target/web-app WA_TOKEN=secret WO_DATA=./data ./target/web-app 8080 +A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. + ## TLS / HTTP2 This sample runs plaintext behind nginx/caddy — the proxy terminates TLS+ALPN diff --git a/docs/guides/deploying-site.md b/docs/guides/deploying-site.md index 3a6a0e8..427c0c6 100644 --- a/docs/guides/deploying-site.md +++ b/docs/guides/deploying-site.md @@ -121,7 +121,7 @@ inside the dependency) is pre-existing and not a build failure. ## Refreshing content — trap 1, in practice Chapter bodies live in `content.wo`, in git. That is their source of truth; -`WO_DATA` is a *replica* seeded on first boot. The site's only table is +`WO_DATA` is a *replica* seeded on first boot (a program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out). The site's only table is `Chapter`, so the WAL holds nothing else — which is what makes the fix safe: ``` diff --git a/docs/guides/log-structured-rows.md b/docs/guides/log-structured-rows.md index 14fbf1d..1e91b88 100644 --- a/docs/guides/log-structured-rows.md +++ b/docs/guides/log-structured-rows.md @@ -44,7 +44,8 @@ Three consequences worth internalising: A `resident: keys` table has no rows in RAM at all, so without a log there is nothing to reconstruct from. That is why the runtime refuses at startup when such a table is declared and `WO_DATA` is unset, rather than letting every read -return "no such row". +return "no such row". Since databasev2 2 task 6a the default table is held to +the same rule: a program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. --- diff --git a/docs/guides/releasing.md b/docs/guides/releasing.md index ed47a3e..472def4 100644 --- a/docs/guides/releasing.md +++ b/docs/guides/releasing.md @@ -75,7 +75,7 @@ ignored by this workflow. Shipping that change to writeonce.de is its own runbook: docs/guides/deploying-site.md. Note especially that a new or renumbered CHAPTER does not appear on a host that already has a - WO_DATA directory — the seed only fills an empty table. + WO_DATA directory — the seed only fills an empty table. A program with any durable table (the default) refuses to start without WO_DATA; WO_EPHEMERAL=1 opts into a RAM-only run, @table(durable: false) opts a table out. docs/examples/site is a SUBMODULE (github.com/shoneyJ/writeonce-site), so this edit is a commit in THAT repo, pushed there, and then a diff --git a/docs/guides/updating-site.md b/docs/guides/updating-site.md index c9b3092..ad95ed9 100644 --- a/docs/guides/updating-site.md +++ b/docs/guides/updating-site.md @@ -60,7 +60,7 @@ proves it renders. What content edits do NOT do is reach a host that already has a `WO_DATA` directory: `seed_if_empty()` fills an empty table only, and the admin route -cannot create a chapter. A new chapter (or renumbered `ord`s) therefore ships +cannot create a chapter. (a program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out.) A new chapter (or renumbered `ord`s) therefore ships with the data-refresh step in [`deploying-site.md`](deploying-site.md#refreshing-content--trap-1-in-practice) — measured there, silent otherwise. diff --git a/docs/stories/databasev2/02-table-storage-modes.md b/docs/stories/databasev2/02-table-storage-modes.md index d26d96d..3ee8e87 100644 --- a/docs/stories/databasev2/02-table-storage-modes.md +++ b/docs/stories/databasev2/02-table-storage-modes.md @@ -1,8 +1,9 @@ --- track: databasev2 iteration: "2" -status: in-progress +status: done readiness: ready +review_pending: "forks 1–7 auto-approved 2026-09-09/10 for autonomous execution — developer second review before close" --- # databasev2 2 — per-table storage: `durable` and `resident` @@ -63,22 +64,27 @@ declared per-table policy. Durability is untouched and unconditional. | 4 | `durable: false` skips the WAL append and replay | ✅ `dd67e31` | | 5a | `wo_wal_next_offset` — exact record offsets | ✅ `ac7d8af` | | 5b | `wo_wal_read_row_at` — a row from a log offset | ✅ `d0c370c` | -| 5c | shared borrow/release accessor, then id→offset storage | ✅ `2e347de` (accessor, pure refactor, `db-bench --quick` 85/0), `18ce4d5` (offset storage), `f9c36ef` (insert + boot wiring) | -| 5d | rewire the readers: remaining `wo_row_ptr` sites, slab scans, FK restrict, `@unique` across the boundary | ✅ `11a92df` (db.c), + this commit (table.c, wal.c, compaction). Updates **refused**, not rewired — see below | -| 6 | the two runtime refusals (no-`WO_DATA`, the byte budget) | ⬜ | -| 7 | measure, gate, document, close out | ✅ measured, gated and documented 2026-08-30 | +| 5c | shared borrow/release accessor, then id→offset storage | ✅ `2e347de` (accessor, pure refactor, `db-bench --quick` 85/0), `125bd09` (offset storage), `08abd09` (insert + boot wiring) — hashes as on `dev`; the pre-merge `18ce4d5`/`f9c36ef` this row used to name are unreachable there | +| 5d | rewire the readers: remaining `wo_row_ptr` sites, slab scans, FK restrict, `@unique` across the boundary | ✅ `0c97fa4` (db.c), `f606fc9` (table.c, wal.c, compaction) — the pre-merge `11a92df` is unreachable on `dev`. Updates were **refused** here, then lifted 2026-08-30 — see below | +| 6a | refuse `durable: true` (the default) with no `WO_DATA`; `WO_EPHEMERAL=1` is the escape hatch | ✅ 2026-09-10 — with the **v8 table bit** (`WO_CLASSF_TABLE`), so the rule applies to `@table` classes only; forks 1–7, see Info | +| 6b | the resident byte budget | ➡ moved to [5](05-bounded-tables-eviction.md) Phase A, 2026-09-09 | +| 7 | measure, gate, document | ✅ `a310496`, 2026-08-30 | **The `durable` half is complete and usable.** A volatile table is a full table in-process — same indexes, same `@unique`, same FK restrict, same query surface — and is simply empty after a restart. That is what -[porch 1–3](../porch/01-store-backed-middleware.md) need for sessions, -rate-limit counters and idempotency keys. +[porch 1–3](../porch/01-store-backed-middleware.md) were written to use for +sessions, rate-limit counters and idempotency keys — though `store.wo` in fact +declares both tables default-durable, which is why fork 6 (below) bites and +the porch gates set `WO_DATA`. **The `resident: keys` half is fully wired for CRUD.** Storage, reads, scans, `@unique`, deletes and updates (a WAL delta record, folded back to a value on -every read) all work, and survive both a restart and a WAL checkpoint. Task -6's two runtime refusals and task 7's measurement are what remain — see -Outstanding below. +every read) all work, and survive both a restart and a WAL checkpoint. Task 7 +measured and gated it on 2026-08-30 (`a310496`). Task 6a — the no-`WO_DATA` +refusal and its `WO_EPHEMERAL=1` escape hatch — landed 2026-09-10, and with it +the iteration closes; the byte budget (6b) moved to +[5](05-bounded-tables-eviction.md) on 2026-09-09. ## Acceptance Criteria @@ -208,6 +214,61 @@ Met: cost becomes at most K+1 reads and replay O(K²) per row, independent of when a checkpoint fires. Limitations 2 and 3 above both fall to it. +- **Given** the `resident: all` read baseline, **when** re-measured, **then** + inside tolerance — no cost for a feature not used. ✅ Verified as a + by-product of the residency leg: `resident: all` is unchanged at 1 354 554 + reads/sec uncapped, and every other db-bench leg still runs, which the + keys-resident classes had briefly broken by forcing `WO_DATA` module-wide. +- **Given** a `resident: keys` table larger than RAM, **when** read randomly, + **then** its read cost is measured against the resident baseline on its own + read path. ✅ Measured 2026-08-30 and the answer is qualified: **1.53×** + faster than letting the kernel swap under a cap that binds one and not the + other — real, but nowhere near iteration 1's 273× swap figure would suggest, + because cgroup limits charge the page cache, so moving rows to a file does + not escape a container memory limit. The unambiguous win is footprint: + **2.55×** smaller resident set. Full method, numbers and the failed first + attempt below; gated by `residency.*`. + +Task 6a — met 2026-09-10 (`scripts/residency-accept.sh` section 7, six checks; +`runtime/test/test_loader.c` `test_storage_flags_need_table`): + +- **Given** a `@table` that is `durable: true` (the default) and no `WO_DATA`, + **when** the program starts, **then** it refuses: exit 2 and ONE stderr line + naming the first default-durable table and all three ways forward — + `WO_DATA=`, `WO_EPHEMERAL=1`, or `@table(durable: false)` on that + class. No "+N more". *(Before, this combination silently discarded every + write.)* +- **Given** a program whose classes carry no `@table` at all, **when** it + starts with no `WO_DATA` and no `WO_EPHEMERAL`, **then** rc 0 and nothing on + stderr — `durable:` is a table property, told apart by the `.wob` v8 table + bit; the loader refuses storage bits on a class without it. +- **Given** `WO_EPHEMERAL=1` and no `WO_DATA`, **when** a program with + default-durable tables starts, **then** it runs (rc 0), prints one boot + notice on stderr saying the sentinel is in force, and every write takes + today's RAM path byte for byte — `db.c`'s guards are untouched. +- **Given** `WO_EPHEMERAL` set together with a non-empty `WO_DATA`, **when** + the program starts, **then** exit 2 with one stderr line naming the + conflict. Any `WO_EPHEMERAL` value other than `1` is the same refusal. +- **Given** `WO_EPHEMERAL=1` and a `resident: keys` class, **when** the program + starts, **then** exit 2 with the EXISTING keys-need-a-log message — the + sentinel does not bypass that loop. +- **Given** the harness after 6a lands, **when** the gates run, **then** the + goldens move at most once, for the `.wob` v8 version byte and the `table` + flag (measured: none moved — the bytecode dump prints flags by name, no + `bc/` golden declares a table, and the header version is not printed); + `just oop-e2e` is green with exactly one harness line changed (the corpus's + `export WO_EPHEMERAL=1`); `db-bench --quick`'s wired RAM legs (`ram`, + `msgrate`, `growth`, `randread` under `WO_EPHEMERAL=1`) sit inside their + floors; every gate that sets `WO_DATA` is unchanged; gates whose programs + declare no `@table` (fibers, subprocess, log-watcher) run untouched; and the + two that turned out to carry durable tables after all opt in by measurement + — chat through porch's store (`RateLimitCounter`, fork 6) and wmux's client + legs, which run the default-durable server image with no `WO_DATA`. + +The resident byte budget (task 6b until 2026-09-09) is no longer this +iteration's: it is [5](05-bounded-tables-eviction.md)'s Phase A, with the +brief's design inputs recorded there as notes for 5's own brainstorm. + ### Task 7 — measured 2026-08-30, and the answer is qualified **The question**, in the words this file has carried since the iteration was @@ -303,28 +364,7 @@ leg did. Verified by feeding the gate a breaching run: `rss_ratio` 1.4, `overcap_vs_swap_x` 0.6 and `in_ram_cost_x` 12.0 are all rejected. -- **Given** the `resident: all` read baseline, **when** re-measured, **then** - inside tolerance — no cost for a feature not used. ✅ Verified as a - by-product of the residency leg: `resident: all` is unchanged at 1 354 554 - reads/sec uncapped, and every other db-bench leg still runs, which the - keys-resident classes had briefly broken by forcing `WO_DATA` module-wide. -- **Given** a `resident: keys` table larger than RAM, **when** read randomly, - **then** its read cost is measured against the resident baseline on its own - read path. ✅ Measured 2026-08-30 and the answer is qualified: **1.53×** - faster than letting the kernel swap under a cap that binds one and not the - other — real, but nowhere near iteration 1's 273× swap figure would suggest, - because cgroup limits charge the page cache, so moving rows to a file does - not escape a container memory limit. The unambiguous win is footprint: - **2.55×** smaller resident set. Full method, numbers and the failed first - attempt above; gated by `residency.*`. -Outstanding: - -- **Given** `durable: true` and no `WO_DATA`, **when** the program starts, - **then** it refuses. *(task 6 — today this combination silently discards - every write)* -- **Given** the resident footprint crossing the budget, **when** it does, - **then** a refusal naming the table and the annotation. *(task 6)* ## Out Of Scope - **Checkpoint and compaction** — [3](03-wal-checkpoint.md). Boot rebuilds the @@ -362,7 +402,67 @@ Outstanding: 5. **The budget is bytes, not rows** — a text-heavy row and an Int-only row differ by 3.3× (measured, databasev2 1), so a row count cannot bound RAM. -## History — two corrections worth keeping +**Task 6a — forks 1–6 locked 2026-09-09, fork 7 added 2026-09-10** (auto-approved +for autonomous execution; the frontmatter's `review_pending` asks for the +developer's second review before close): + +1. **The escape hatch is the environment sentinel `WO_EPHEMERAL=1`, exact + value.** Not `WO_DATA=:memory:` — `WO_DATA` stays a path and only a path, + which also keeps [7](07-single-file-db.md)'s file-vs-directory parse free + of sentinels — and no fixture edits. `@table(durable: false)` remains the + per-table declaration; the sentinel is the whole-program one. +2. **Sentinel rules.** Honoured only when `WO_DATA` is unset or empty. + `WO_EPHEMERAL` set together with `WO_DATA` is a startup refusal, exit 2, + naming the conflict. Any value other than `1` is the same refusal. A + `resident: keys` class still refuses through the existing loop — the + sentinel does not bypass it. One stderr boot notice when the sentinel is + in force. +3. **The refusal contract** for `durable: true` (the default) with no + `WO_DATA`: exit 2, one stderr line naming the first default-durable class + and all three ways forward — `WO_DATA=`, `WO_EPHEMERAL=1`, + `@table(durable: false)`. A second loop in `runtime/src/main.c`'s existing + startup-refusal block, beside the `resident: keys` one. No "+N more". +4. **Startup-only.** `db.c`'s guards are untouched, so the ephemeral path is + byte for byte today's RAM path; nothing on the data path learns a flag. +5. **The byte budget (6b) leaves this iteration** for + [5](05-bounded-tables-eviction.md), as a new Phase A — a per-table + resident byte counter feeding its pressure signal. 5 stays + `readiness: refine`; the design inputs go there as notes for its own + brainstorm, and the spec's three budget obligations become 5's acceptance + criteria. +6. **Library-owned durable tables bind every consumer.** + [porch's store](../../examples/porch/middleware/store.wo) declares its + tables with the default, so any program that `use`s it needs `WO_DATA` or + `WO_EPHEMERAL=1` as a whole-program requirement. No consumer-side + override: the library author's declaration is the declaration. Settled. +7. **The table bit in the `.wob` (v8), so durability rules apply to `@table` + classes only** — added 2026-09-10 after the first cut refused every + class-bearing program (fibers' `Tick`, subprocess's `ConnMsg`, log-watcher's + `CronEntry` are plain classes, and v7 spelled `durable: true` as the mere + absence of the volatile bit). `WO_CLASSF_TABLE` 0x08 is set from the + emitter's `cr_is_table`; the loader refuses the two storage bits without + it; `main.c`'s two refusal loops skip classes without it; a program with + no durable table does not consult `WO_EPHEMERAL` at all (the + `WO_DATA`+`WO_EPHEMERAL` conflict still refuses regardless). A v7 image is + refused by the version check, as v6 was by v7. The alternative — teaching + the runtime to infer "table" from the presence of indexes or an `insert` + site — was rejected: a fact the compiler already holds belongs in the + image, not re-derived. + +## History — three corrections worth keeping + +**Task 6a's first cut refused every class-bearing program (2026-09-09→10).** +The refusal keyed on "flags lack `WO_CLASSF_VOLATILE`", and the v7 image had +no bit saying "this class is a `@table`" — so `class Tick` in fibers looked +exactly like a default-durable table, and gates that never touch a table +(fibers, subprocess, log-watcher) had to export `WO_EPHEMERAL=1` to start. +Corrected the next day by recording the missing fact in the image (`.wob` v8, +`WO_CLASSF_TABLE`, fork 7), then re-measuring every gate without its export +and keeping the sentinel only where the program really refused: the corpus, +db-bench and db-actor (their own tables), chat (porch's store, fork 6) and +wmux's client legs (the server image, no `WO_DATA`). The lesson: "does this +gate run a table program" is answered by running it, not by reading the +example's own source — a `use`d library's declaration counts. **The three-mode design was replaced.** Earlier drafts had `mode: ram | durable | cold`. `cold` conflated two independent properties and @@ -381,7 +481,11 @@ fine, the plan's storage steps still read as plumbing. Measured instead: `wo_row_ptr` returns a `db_row *` into a slab and has 11 call sites, `table.c` has 37 slab references, `db.c:105-181` walks slabs for scans, `enc_val` serialises *from* the slab, and **no operation exists that drops a row's payload -while keeping its index entries**. Hence the 5a–5d split. 5c and 5d need their -own write-ups, and the two open design questions for 5c are whether the id hash -stores offsets in place of slot indices or gains a parallel map, and what the -new operation does about the unique shadows, which currently point at slots. +while keeping its index entries**. Hence the 5a–5d split. Both 5c questions +are settled by what landed (2026-08-29): the id hash stores the LOG OFFSET in +place of the slot index — one map, no parallel one — which is also why the +`delete` corruption above was possible, since a reader that trusts that value +as a slot indexes a slab with a byte offset; and the unique shadows keep their +bucket candidates and resolve them through the same borrow, one `pread` per +candidate, never a slot dereference. The write-ups are the 5c/5d acceptance +bullets above and the 2026-08-29/30 board entries. diff --git a/docs/superpowers/plans/2026-08-26-table-residency.md b/docs/superpowers/plans/2026-08-26-table-residency.md index 29d3efe..72b4ed0 100644 --- a/docs/superpowers/plans/2026-08-26-table-residency.md +++ b/docs/superpowers/plans/2026-08-26-table-residency.md @@ -385,25 +385,36 @@ plus wherever per-table accounting lands from Task 6. so a program that declares durability and is given nowhere to put it silently loses everything. Refuse at startup, naming the first durable class found. This is the single most valuable line in the plan and is independent of - residency. + residency. *Decided 2026-09-09: exit 2, one stderr line, the first + default-durable class and all three ways forward; a second loop in `main.c`'s + existing startup-refusal block.* - [ ] Provide the escape hatch the refusal implies: a program that genuinely wants an ephemeral run must be able to say so, either by declaring its tables volatile or by an explicit opt-out flag. Decide which and document it — a refusal with no stated way forward is a worse bug than the silent loss. + *Decided 2026-09-09: `WO_EPHEMERAL=1`, exact value, honoured only when + `WO_DATA` is unset or empty; set together is a refusal; any other value is a + refusal; `@table(durable: false)` stays the per-table form.* - [ ] Implement the byte budget: estimated resident footprint across all tables, breached loudly with a message naming the largest offending table and - the exact annotation to add. + the exact annotation to add. *Moved to databasev2 5 Phase A (2026-09-09).* - [ ] Default the budget to a fraction of host-detected available memory, **not to "none"** — a budget nobody sets cannot produce the diagnostic that is this design's main deliverable, and the 120 GB developer would still meet the OOM killer. Use a conservative placeholder fraction and mark the value explicitly unset-pending in both the code comment and the iteration: the real number - comes from databasev2 1's swap-onset measurement. + comes from databasev2 1's swap-onset measurement. *Moved to databasev2 5 + (2026-09-09) as a note for its brainstorm: bytes XOR fraction, + `MemAvailable` and the cgroup v2 limits as the denominator, an itemised + measured reserve with a floor. Databasev2 1 found there is no swap onset to + derive from.* - [ ] Make the accounting's error bound explicit where it is documented. It estimates RSS; it is not RSS, and pretending otherwise would make the budget - untrustworthy the first time someone checked it. + untrustworthy the first time someone checked it. *Moved to databasev2 5 + (2026-09-09) — an acceptance criterion there.* - [ ] Add CLI-smoke coverage for both refusals, including the exit code and the - first line of stderr — the shape scripts depend on. + first line of stderr — the shape scripts depend on. *2026-09-09: one refusal + here now (6a); the budget's is 5's.* - [ ] Verify: `bash runtime/test/cli_smoke.sh`, `just wovm-test`, `just employee`, `just db-actor` green; every sample that sets `WO_DATA` still runs, and one that does not is now refused or explicitly opted out. diff --git a/docs/superpowers/specs/2026-08-26-table-residency-design.md b/docs/superpowers/specs/2026-08-26-table-residency-design.md index b702f68..47ddfc4 100644 --- a/docs/superpowers/specs/2026-08-26-table-residency-design.md +++ b/docs/superpowers/specs/2026-08-26-table-residency-design.md @@ -215,8 +215,14 @@ as the code — not afterwards: combination silently loses everything, which is the worst failure mode in the current engine. A program that declares durability and is given nowhere to put it must not start. This is independent of residency and is arguably the most - valuable single line in the spec. -- **A byte budget that exists by default, breached loudly.** The budget bounds + valuable single line in the spec. *Decided 2026-09-09 (databasev2 2 task 6a): + exit 2, one stderr line naming the first default-durable class and the three + ways forward — `WO_DATA=`, `WO_EPHEMERAL=1`, `@table(durable: false)`; + the whole-program escape hatch is `WO_EPHEMERAL=1`, exact value, honoured + only when `WO_DATA` is unset.* +- **A byte budget that exists by default, breached loudly.** *Moved to + [databasev2 5](../../stories/databasev2/05-bounded-tables-eviction.md) Phase A + on 2026-09-09; the text below stands as the original obligation.* The budget bounds estimated resident footprint across all tables. On breach the program refuses with a message naming the largest offending table and the exact annotation to add, so the 120 GB developer meets a diagnostic at 32 GB rather than the OOM @@ -298,7 +304,9 @@ Acceptance is the story's Given/When/Then list; this is how each is exercised. - **Compiler refusals** — a `compile-fail` fixture per diagnostic, each pinning the exact code. - **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup; - a budget breach names the table and the annotation. + a budget breach names the table and the annotation. *(2026-09-09: the first + is task 6a, escape hatch `WO_EPHEMERAL=1`; the budget breach moved to + databasev2 5.)* - **Crash safety** — `kill -9` mid-append and mid-checkpoint on a `resident: keys` table; replay loses no acked write and no row appears twice. - **Performance** — new baseline rows for the `resident: keys` read path with @@ -337,10 +345,13 @@ unchanged. loud one, and both listed here so neither arrives as a surprise: 1. `durable: true` (the default) with no `WO_DATA` becomes a startup refusal. - Today it silently discards every write. + Today it silently discards every write. *(Decided 2026-09-09: + `WO_EPHEMERAL=1` opts the whole program out.)* 2. Total estimated resident footprint crossing the default budget fraction becomes a startup or insert refusal. Today the program slides into swap with - no signal and is eventually killed. + no signal and is eventually killed. *(Moved to databasev2 5 on 2026-09-09 — + no longer part of iteration 2's behaviour change.)* -Both are opt-out-able by explicit declaration. Neither is a data-format change, +Both are opt-out-able by explicit declaration *(for the first: per table by +`@table(durable: false)`, or for the program by `WO_EPHEMERAL=1`)*. Neither is a data-format change, so a rollback is a binary swap with no migration.