docs(db2-ephemeral): databasev2 2 closes — task 6a contract, forks 1–7, the README sweep

- story 02: `status: done`, `review_pending` (forks 1–7 auto-approved for
  autonomy); progress rows 6a ✅, 6b ➡ databasev2 5 Phase A, 7 `a310496`;
  5c/5d rows cite the `dev` hashes (the pre-merge ones were unreachable);
  task 6a's Given/When/Then met; Info records the seven forks (sentinel over
  `:memory:`, its rules, the refusal contract, startup-only, the budget
  leaves for 5, library-owned tables bind consumers, the v8 table bit);
  History keeps the first cut that refused every class-bearing program
- database/src/CODE-LOGIC.md: "Startup refusal + WO_EPHEMERAL" — contract,
  hatch, table bit, measured blast radius, deferred items, proof; the
  dispatcher paragraph no longer says a failed commit un-applies the row
  (fatal since databasev2 4 part A; WO_T_IO unreachable from a write path)
- residency spec + plan: task 6 items annotated with the 2026-09-09
  decisions; the byte budget marked moved to databasev2 5
- README, seven example READMEs and four guides carry the one-line rule
  (durable default refuses without WO_DATA; WO_EPHEMERAL=1; durable:
  false); shop's RAM-only command sets the sentinel

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 2c3531998124042fe736388e8b926abda3841194)
This commit is contained in:
shoney.arickathil 2026-09-15 01:03:49 +02:00
parent 41f48bba3f
commit 296efb52ed
16 changed files with 266 additions and 61 deletions

View file

@ -302,6 +302,8 @@ WO_DATA=./data ./target/myproject seed
WO_DATA=./data ./target/myproject report # a fresh process still sees the data WO_DATA=./data ./target/myproject report # a fresh process still sees the data
``` ```
A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out.
--- ---
## Worked examples ## Worked examples

View file

@ -84,11 +84,15 @@ succeed and the file is the only artifact beside a decoy sibling directory).
One dispatcher, the builtin contract (0 ok, else WO_T_* + msg). The engine One dispatcher, the builtin contract (0 ok, else WO_T_* + msg). The engine
handles ride `wo_rt.db` / `wo_rt.wal` as opaque pointers set by main.c — handles ride `wo_rt.db` / `wo_rt.wal` as opaque pointers set by main.c —
NULL db traps WO_T_DB, NULL wal means RAM-only (the corpus's mode; WO_DATA NULL db traps WO_T_DB, NULL wal means RAM-only — reachable only under
opts into durability). Insert's contract: RAM apply through the row API, WO_EPHEMERAL=1 (or with no durable `@table` in the module) since databasev2 2
then stage + commit BEFORE returning — the builtin's return is the task 6a: a program with any durable `@table` (the default) refuses to start
acknowledgment, so a failed commit un-applies the row and traps WO_T_IO without WO_DATA; WO_EPHEMERAL=1 opts into a RAM-only run (the corpus's mode),
rather than acknowledging what disk never got. @table(durable: false) opts a table out. Insert's contract: RAM apply through
the row API, then stage + commit BEFORE returning — the builtin's return is
the acknowledgment. Once RAM has mutated the outcomes are durable or process
death (`wo_wal_commit_fatal`, `wo_wal_stage_fatal`): a failed commit is
fatal, there is no un-apply, and WO_T_IO is unreachable from a write path.
## Verifying a change ## Verifying a change
@ -400,3 +404,65 @@ A `@table` class is the schema; the log is the database; boot compares them.
- **Legacy logs** (no head record) replay exactly as before and adopt the - **Legacy logs** (no head record) replay exactly as before and adopt the
head at their next compaction. v1 verbs are add and delete only; rename head at their next compaction. v1 verbs are add and delete only; rename
wants `@renamed_from` (v2), data/seed migrations are v2. wants `@renamed_from` (v2), data/seed migrations are v2.
## Startup refusal + WO_EPHEMERAL (databasev2 2 task 6a, 2026-09-09/10)
`main.c`, startup only. The engine, `db.c` and `wal.c` are untouched.
- **Contract.** With `WO_DATA` unset or empty and no `WO_EPHEMERAL`, the first
class whose flags carry `WO_CLASSF_TABLE` and lack `WO_CLASSF_VOLATILE`
(a `@table` with `durable: true`, the default) is a startup refusal: exit 2,
ONE stderr line naming the class and all three ways forward literally
(`WO_DATA=<dir or file>`, `WO_EPHEMERAL=1`, `@table(durable: false)`). The loop sits
inside the existing `!data_dir` block AFTER the `resident: keys` loop — the
keys refusal wins, and `WO_EPHEMERAL` does not rescue it (a keys table has
nowhere to read from). Both loops skip classes without the table bit.
- **Escape hatch.** `WO_EPHEMERAL` with the exact value `1`, honoured only
while `WO_DATA` is unset/empty: one boot notice line on stderr, rc 0, and the
RAM path is byte-for-byte the old one — `db.c`'s `w && table_is_durable`
guards are the only gate, no new flag in `wo_db`. Set alongside `WO_DATA`
(any value) → exit 2 `WO_EPHEMERAL=1 is incompatible with WO_DATA` — that
check runs regardless of tables. Any value but `1` → exit 2 naming the
accepted value. A module with no durable `@table` consults `WO_EPHEMERAL`
for nothing else: no notice, no value check, rc 0 as before.
- **The table bit (`.wob` v8, 2026-09-10).** The first cut keyed the refusal
on `!VOLATILE` alone, and the v7 image carried no "is a `@table`" bit: plain
classes, variant classes and the predeclared records (`Error`, `Stat`, …)
all looked durable, so EVERY class-bearing program refused without
`WO_DATA` — fibers (`Tick`), subprocess (`ConnMsg`), log-watcher
(`CronEntry`), chat. Wrong by construction: `durable:` is a `@table`
property. Fixed by `WO_CLASSF_TABLE` 0x08 (`wob.h`, `WO_CLASSF_ALL` 0x0f,
`WOB_VERSION` 8; `emit.ml` sets it from `cr_is_table`); the loader refuses
`VOLATILE`/`RESIDENT_KEYS` without it ("storage flags on a class that is
not a @table", `test_loader`), and a v7 image is refused by the version
check exactly as v7 refused v6. Blast radius after the fix, measured gate
by gate (each run without the export first; kept only where it refused):
only programs that DECLARE a durable table opt in — `oop-e2e.sh` (corpus
fixtures declare tables); `db-bench.py`'s ram/msgrate/growth/randread legs
(db-bench's tables); db-actor per-run (`notes` is default-durable; per-run
because its restart pair sets `WO_DATA` and the two are incompatible),
whose single-shard byte-exact compare drops the one notice line
(`grep -v '^wovm: WO_EPHEMERAL=1'`); chat, whose program declares no table
itself but `use`s porch, and porch's store middleware declares
`RateLimitCounter` default-durable — fork 6, a library-owned table binds
the consumer; and wmux, whose CLIENT legs (ls/new/attach/kill) run the
same default-durable image with no `WO_DATA` — the gate exports the
sentinel, every server start and the `WO_DATA`-carrying `r11cli` drop it
with `env -u`, and `client()` filters the notice because its answers are
compared byte-exactly (a wmux-track consequence worth its own look: a CLI
client of a durable server now needs the sentinel or a `WO_DATA`). fibers
(`Tick`), subprocess (`ConnMsg`) and log-watcher (`CronEntry`) need
nothing — their exports were reverted and their byte-exact compares are
as they were. Goldens: none moved — the bytecode dump prints flags by name
and no `bc/` golden declares a table; the header version is not printed.
- **Deferred, each its own later commit:** an assert in `db.c` that
`durable && !w` is unreachable outside `WO_EPHEMERAL`; an ENOENT hint when
the `WO_DATA` directory is missing (the FILE form already refuses with a
named parent since databasev2 7; the directory form still fails at the WAL
open, on purpose — byte-identical to before); SIGKILL /
rc 137 classification in the gates; `wal.c` fallocate/dir-fsync logging.
- **Proof:** `scripts/residency-accept.sh` section 7 — refusal text, RAM
round-trip under the hatch, the `WO_DATA` conflict, keys still refusing
under the hatch, a non-`1` value, and (vi) a plain class without `@table`
running with no `WO_DATA` and nothing on stderr (the corpus `methods`
fixture); `runtime/test/test_loader.c` `test_storage_flags_need_table`.

View file

@ -43,7 +43,7 @@ WO_SHARDS=1 ./docs/examples/db-actor/target/db-actor # force the local path
| multi-shard, three rounds | The writer lines are asserted as a **set**, not a sequence — scheduling decides their order, and pinning it would be testing the scheduler, not the RPC. The `main` line is exact. | | multi-shard, three rounds | The writer lines are asserted as a **set**, not a sequence — scheduling decides their order, and pinning it would be testing the scheduler, not the RPC. The `main` line is exact. |
| both `WO_IO` backends forced | The reply park has to be plane-independent: io_uring and epoll must give the same answer, or the parking is leaking into semantics. | | both `WO_IO` backends forced | The reply park has to be plane-independent: io_uring and epoll must give the same answer, or the parking is leaking into semantics. |
| single shard, byte-exact | The local path is untouched by stage 3. Any drift here means the RPC changed the non-RPC case. | | single shard, byte-exact | The local path is untouched by stage 3. Any drift here means the RPC changed the non-RPC case. |
| `WO_DATA` restart pair | A worker's insert must commit on the **owner's** WAL before its ack, so a restart replays it: 2 rows, then 2+2 after a second run. This is the durability claim the RPC could most easily break. | | `WO_DATA` restart pair | A worker's insert must commit on the **owner's** WAL before its ack, so a restart replays it: 2 rows, then 2+2 after a second run. This is the durability claim the RPC could most easily break. A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. |
Run under `wovm_asan` and `wovm_tsan` as well — cross-shard message passing is Run under `wovm_asan` and `wovm_tsan` as well — cross-shard message passing is
exactly where a data race would hide, and TSan covering this demo is the one exactly where a data race would hide, and TSan covering this demo is the one

View file

@ -33,7 +33,8 @@ strictly better. Recorded as a plan deviation.)
| var | effect | | var | effect |
| --- | --- | | --- | --- |
| `WO_DATA=<dir>` | durability on: replay `<dir>/shard-0.wal` at boot, log every write. Without it the store is RAM-only | | `WO_DATA=<dir>` (or `WO_DATA=<path>.db`, below) | durability on: replay `<dir>/shard-0.wal` at boot, log every write. A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. |
| `WO_EPHEMERAL=1` | **databasev2 2 task 6a:** the RAM-only opt-in the driver sets on its ram/msgrate/growth/randread legs. A `WO_DATA` exported in your shell no longer silently turns those legs durable — the two are incompatible and the run refuses loudly |
| `WO_SHARDS=<n>` | shard count. **`1` means every statement runs inline on shard 0 and group commit cannot engage** — batches form only where writes queue from other shards | | `WO_SHARDS=<n>` | shard count. **`1` means every statement runs inline on shard 0 and group commit cannot engage** — batches form only where writes queue from other shards |
| `WO_CHECKPOINT_BYTES` / `WO_CHECKPOINT_RATIO` | **databasev2 3:** the checkpoint trigger — the log must exceed the floor AND exceed the ratio times the last compaction's own size. A tiny floor forces compaction in a few writes, which is how the gate tests the policy at all; an enormous one disables it, which is how the checkpoint leg measures the same workload with and without | | `WO_CHECKPOINT_BYTES` / `WO_CHECKPOINT_RATIO` | **databasev2 3:** the checkpoint trigger — the log must exceed the floor AND exceed the ratio times the last compaction's own size. A tiny floor forces compaction in a few writes, which is how the gate tests the policy at all; an enormous one disables it, which is how the checkpoint leg measures the same workload with and without |
| `WO_WAL_STATS=1` | **databasev2 4:** print one line at exit — `walstats batches=… records=… peak_batch=… peak_staged=… compactions=… compact_us_max=… compact_us_total=… compacted_bytes=…`. Opt-in so it does not pollute every durable program's output. Mean batch is `records/batches`; **mean 1.0 means group commit is not engaging**, which is expected for a serial writer or `WO_SHARDS=1` and a bug anywhere else | | `WO_WAL_STATS=1` | **databasev2 4:** print one line at exit — `walstats batches=… records=… peak_batch=… peak_staged=… compactions=… compact_us_max=… compact_us_total=… compacted_bytes=…`. Opt-in so it does not pollute every durable program's output. Mean batch is `records/batches`; **mean 1.0 means group commit is not engaging**, which is expected for a serial writer or `WO_SHARDS=1` and a bug anywhere else |

View file

@ -30,6 +30,11 @@ and pasted — the `PASTE-…-HERE` placeholders mark exactly where. The
connect-section name is the code's namespace: `[connect.employee]` is why connect-section name is the code's namespace: `[connect.employee]` is why
the source says `employee.Employee`. the source says `employee.Employee`.
B declares the shapes but stores nothing, so it runs under `WO_EPHEMERAL=1`: a
program with any durable table (the default) refuses to start without
`WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)`
opts a table out.
| Mode | What it proves | | Mode | What it proves |
| --- | --- | | --- | --- |
| `employee-list list` | typed reads over the wire, `e.dept.name` ref navigation executing inside A | | `employee-list list` | typed reads over the wire, `e.dept.name` ref navigation executing inside A |

View file

@ -8,7 +8,7 @@ Before it, `WO_DATA` was the only switch. Set, and every table is WAL-logged;
unset, and none are. Applications are not uniform — a session table is unset, and none are. Applications are not uniform — a session table is
disposable, a product's stock level is not, and a catalogue large enough to disposable, a product's stock level is not, and a catalogue large enough to
matter does not fit in RAM at all. One global switch forces "everything is precious" or "nothing matter does not fit in RAM at all. One global switch forces "everything is precious" or "nothing
is", and you pay for whichever is wrong. is", and you pay for whichever is wrong. Since task 6a the default is enforced: a program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out.
## Run it ## Run it

View file

@ -10,9 +10,11 @@ has one concern, and the module system (one directory = one module,
``` ```
cd docs/examples/shop cd docs/examples/shop
woc . && WO_DATA=./data ./target/shop 8080 # durable store woc . && WO_DATA=./data ./target/shop 8080 # durable store
./target/shop 8080 # RAM-only (dev) WO_EPHEMERAL=1 ./target/shop 8080 # RAM-only (dev)
``` ```
A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out.
Browse http://127.0.0.1:8080/ — products → product page → buy (stock Browse http://127.0.0.1:8080/ — products → product page → buy (stock
checked and decremented) → confirmation → /orders. With `WO_DATA`, kill checked and decremented) → confirmation → /orders. With `WO_DATA`, kill
it and restart: the orders are still there (WAL replay). it and restart: the orders are still there (WAL replay).

View file

@ -27,4 +27,4 @@ subquery construct is needed; the general `exists`/`not exists` is deferred
until a corpus uses a correlation a backlink cannot express. until a corpus uses a correlation a backlink cannot express.
`skill-catalog seed | list | roots | count | get <name>`, WAL-durable under `skill-catalog seed | list | roots | count | get <name>`, WAL-durable under
`WO_DATA`. Acceptance: `scripts/skill-catalog-accept.sh`. `WO_DATA`. A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out. Acceptance: `scripts/skill-catalog-accept.sh`.

View file

@ -24,6 +24,8 @@ the token comes from the `WA_TOKEN` env var.
woc . # fetches deps, builds target/web-app woc . # fetches deps, builds target/web-app
WA_TOKEN=secret WO_DATA=./data ./target/web-app 8080 WA_TOKEN=secret WO_DATA=./data ./target/web-app 8080
A program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out.
## TLS / HTTP2 ## TLS / HTTP2
This sample runs plaintext behind nginx/caddy — the proxy terminates TLS+ALPN This sample runs plaintext behind nginx/caddy — the proxy terminates TLS+ALPN

View file

@ -121,7 +121,7 @@ inside the dependency) is pre-existing and not a build failure.
## Refreshing content — trap 1, in practice ## Refreshing content — trap 1, in practice
Chapter bodies live in `content.wo`, in git. That is their source of truth; Chapter bodies live in `content.wo`, in git. That is their source of truth;
`WO_DATA` is a *replica* seeded on first boot. The site's only table is `WO_DATA` is a *replica* seeded on first boot (a program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out). The site's only table is
`Chapter`, so the WAL holds nothing else — which is what makes the fix safe: `Chapter`, so the WAL holds nothing else — which is what makes the fix safe:
``` ```

View file

@ -44,7 +44,8 @@ Three consequences worth internalising:
A `resident: keys` table has no rows in RAM at all, so without a log there is A `resident: keys` table has no rows in RAM at all, so without a log there is
nothing to reconstruct from. That is why the runtime refuses at startup when nothing to reconstruct from. That is why the runtime refuses at startup when
such a table is declared and `WO_DATA` is unset, rather than letting every read such a table is declared and `WO_DATA` is unset, rather than letting every read
return "no such row". return "no such row". Since databasev2 2 task 6a the default table is held to
the same rule: a program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out.
--- ---

View file

@ -75,7 +75,7 @@ ignored by this workflow.
Shipping that change to writeonce.de is its own runbook: Shipping that change to writeonce.de is its own runbook:
docs/guides/deploying-site.md. Note especially that a new or docs/guides/deploying-site.md. Note especially that a new or
renumbered CHAPTER does not appear on a host that already has a renumbered CHAPTER does not appear on a host that already has a
WO_DATA directory — the seed only fills an empty table. WO_DATA directory — the seed only fills an empty table. A program with any durable table (the default) refuses to start without WO_DATA; WO_EPHEMERAL=1 opts into a RAM-only run, @table(durable: false) opts a table out.
docs/examples/site is a SUBMODULE (github.com/shoneyJ/writeonce-site), docs/examples/site is a SUBMODULE (github.com/shoneyJ/writeonce-site),
so this edit is a commit in THAT repo, pushed there, and then a so this edit is a commit in THAT repo, pushed there, and then a

View file

@ -60,7 +60,7 @@ proves it renders.
What content edits do NOT do is reach a host that already has a `WO_DATA` What content edits do NOT do is reach a host that already has a `WO_DATA`
directory: `seed_if_empty()` fills an empty table only, and the admin route directory: `seed_if_empty()` fills an empty table only, and the admin route
cannot create a chapter. A new chapter (or renumbered `ord`s) therefore ships cannot create a chapter. (a program with any durable table (the default) refuses to start without `WO_DATA`; `WO_EPHEMERAL=1` opts into a RAM-only run, `@table(durable: false)` opts a table out.) A new chapter (or renumbered `ord`s) therefore ships
with the data-refresh step in with the data-refresh step in
[`deploying-site.md`](deploying-site.md#refreshing-content--trap-1-in-practice) [`deploying-site.md`](deploying-site.md#refreshing-content--trap-1-in-practice)
— measured there, silent otherwise. — measured there, silent otherwise.

View file

@ -1,8 +1,9 @@
--- ---
track: databasev2 track: databasev2
iteration: "2" iteration: "2"
status: in-progress status: done
readiness: ready readiness: ready
review_pending: "forks 1–7 auto-approved 2026-09-09/10 for autonomous execution — developer second review before close"
--- ---
# databasev2 2 — per-table storage: `durable` and `resident` # databasev2 2 — per-table storage: `durable` and `resident`
@ -63,22 +64,27 @@ declared per-table policy. Durability is untouched and unconditional.
| 4 | `durable: false` skips the WAL append and replay | ✅ `dd67e31` | | 4 | `durable: false` skips the WAL append and replay | ✅ `dd67e31` |
| 5a | `wo_wal_next_offset` — exact record offsets | ✅ `ac7d8af` | | 5a | `wo_wal_next_offset` — exact record offsets | ✅ `ac7d8af` |
| 5b | `wo_wal_read_row_at` — a row from a log offset | ✅ `d0c370c` | | 5b | `wo_wal_read_row_at` — a row from a log offset | ✅ `d0c370c` |
| 5c | shared borrow/release accessor, then id→offset storage | ✅ `2e347de` (accessor, pure refactor, `db-bench --quick` 85/0), `18ce4d5` (offset storage), `f9c36ef` (insert + boot wiring) | | 5c | shared borrow/release accessor, then id→offset storage | ✅ `2e347de` (accessor, pure refactor, `db-bench --quick` 85/0), `125bd09` (offset storage), `08abd09` (insert + boot wiring) — hashes as on `dev`; the pre-merge `18ce4d5`/`f9c36ef` this row used to name are unreachable there |
| 5d | rewire the readers: remaining `wo_row_ptr` sites, slab scans, FK restrict, `@unique` across the boundary | ✅ `11a92df` (db.c), + this commit (table.c, wal.c, compaction). Updates **refused**, not rewired — see below | | 5d | rewire the readers: remaining `wo_row_ptr` sites, slab scans, FK restrict, `@unique` across the boundary | ✅ `0c97fa4` (db.c), `f606fc9` (table.c, wal.c, compaction) — the pre-merge `11a92df` is unreachable on `dev`. Updates were **refused** here, then lifted 2026-08-30 — see below |
| 6 | the two runtime refusals (no-`WO_DATA`, the byte budget) | ⬜ | | 6a | refuse `durable: true` (the default) with no `WO_DATA`; `WO_EPHEMERAL=1` is the escape hatch | ✅ 2026-09-10 — with the **v8 table bit** (`WO_CLASSF_TABLE`), so the rule applies to `@table` classes only; forks 1–7, see Info |
| 7 | measure, gate, document, close out | ✅ measured, gated and documented 2026-08-30 | | 6b | the resident byte budget | ➡ moved to [5](05-bounded-tables-eviction.md) Phase A, 2026-09-09 |
| 7 | measure, gate, document | ✅ `a310496`, 2026-08-30 |
**The `durable` half is complete and usable.** A volatile table is a full table **The `durable` half is complete and usable.** A volatile table is a full table
in-process — same indexes, same `@unique`, same FK restrict, same query surface in-process — same indexes, same `@unique`, same FK restrict, same query surface
— and is simply empty after a restart. That is what — and is simply empty after a restart. That is what
[porch 1–3](../porch/01-store-backed-middleware.md) need for sessions, [porch 1–3](../porch/01-store-backed-middleware.md) were written to use for
rate-limit counters and idempotency keys. sessions, rate-limit counters and idempotency keys — though `store.wo` in fact
declares both tables default-durable, which is why fork 6 (below) bites and
the porch gates set `WO_DATA`.
**The `resident: keys` half is fully wired for CRUD.** Storage, reads, scans, **The `resident: keys` half is fully wired for CRUD.** Storage, reads, scans,
`@unique`, deletes and updates (a WAL delta record, folded back to a value on `@unique`, deletes and updates (a WAL delta record, folded back to a value on
every read) all work, and survive both a restart and a WAL checkpoint. Task every read) all work, and survive both a restart and a WAL checkpoint. Task 7
6's two runtime refusals and task 7's measurement are what remain — see measured and gated it on 2026-08-30 (`a310496`). Task 6a — the no-`WO_DATA`
Outstanding below. refusal and its `WO_EPHEMERAL=1` escape hatch — landed 2026-09-10, and with it
the iteration closes; the byte budget (6b) moved to
[5](05-bounded-tables-eviction.md) on 2026-09-09.
## Acceptance Criteria ## Acceptance Criteria
@ -208,6 +214,61 @@ Met:
cost becomes at most K+1 reads and replay O(K²) per row, independent of cost becomes at most K+1 reads and replay O(K²) per row, independent of
when a checkpoint fires. Limitations 2 and 3 above both fall to it. when a checkpoint fires. Limitations 2 and 3 above both fall to it.
- **Given** the `resident: all` read baseline, **when** re-measured, **then**
inside tolerance — no cost for a feature not used. ✅ Verified as a
by-product of the residency leg: `resident: all` is unchanged at 1 354 554
reads/sec uncapped, and every other db-bench leg still runs, which the
keys-resident classes had briefly broken by forcing `WO_DATA` module-wide.
- **Given** a `resident: keys` table larger than RAM, **when** read randomly,
**then** its read cost is measured against the resident baseline on its own
read path. ✅ Measured 2026-08-30 and the answer is qualified: **1.53×**
faster than letting the kernel swap under a cap that binds one and not the
other — real, but nowhere near iteration 1's 273× swap figure would suggest,
because cgroup limits charge the page cache, so moving rows to a file does
not escape a container memory limit. The unambiguous win is footprint:
**2.55×** smaller resident set. Full method, numbers and the failed first
attempt below; gated by `residency.*`.
Task 6a — met 2026-09-10 (`scripts/residency-accept.sh` section 7, six checks;
`runtime/test/test_loader.c` `test_storage_flags_need_table`):
- **Given** a `@table` that is `durable: true` (the default) and no `WO_DATA`,
**when** the program starts, **then** it refuses: exit 2 and ONE stderr line
naming the first default-durable table and all three ways forward —
`WO_DATA=<dir>`, `WO_EPHEMERAL=1`, or `@table(durable: false)` on that
class. No "+N more". *(Before, this combination silently discarded every
write.)*
- **Given** a program whose classes carry no `@table` at all, **when** it
starts with no `WO_DATA` and no `WO_EPHEMERAL`, **then** rc 0 and nothing on
stderr — `durable:` is a table property, told apart by the `.wob` v8 table
bit; the loader refuses storage bits on a class without it.
- **Given** `WO_EPHEMERAL=1` and no `WO_DATA`, **when** a program with
default-durable tables starts, **then** it runs (rc 0), prints one boot
notice on stderr saying the sentinel is in force, and every write takes
today's RAM path byte for byte — `db.c`'s guards are untouched.
- **Given** `WO_EPHEMERAL` set together with a non-empty `WO_DATA`, **when**
the program starts, **then** exit 2 with one stderr line naming the
conflict. Any `WO_EPHEMERAL` value other than `1` is the same refusal.
- **Given** `WO_EPHEMERAL=1` and a `resident: keys` class, **when** the program
starts, **then** exit 2 with the EXISTING keys-need-a-log message — the
sentinel does not bypass that loop.
- **Given** the harness after 6a lands, **when** the gates run, **then** the
goldens move at most once, for the `.wob` v8 version byte and the `table`
flag (measured: none moved — the bytecode dump prints flags by name, no
`bc/` golden declares a table, and the header version is not printed);
`just oop-e2e` is green with exactly one harness line changed (the corpus's
`export WO_EPHEMERAL=1`); `db-bench --quick`'s wired RAM legs (`ram`,
`msgrate`, `growth`, `randread` under `WO_EPHEMERAL=1`) sit inside their
floors; every gate that sets `WO_DATA` is unchanged; gates whose programs
declare no `@table` (fibers, subprocess, log-watcher) run untouched; and the
two that turned out to carry durable tables after all opt in by measurement
— chat through porch's store (`RateLimitCounter`, fork 6) and wmux's client
legs, which run the default-durable server image with no `WO_DATA`.
The resident byte budget (task 6b until 2026-09-09) is no longer this
iteration's: it is [5](05-bounded-tables-eviction.md)'s Phase A, with the
brief's design inputs recorded there as notes for 5's own brainstorm.
### Task 7 — measured 2026-08-30, and the answer is qualified ### Task 7 — measured 2026-08-30, and the answer is qualified
**The question**, in the words this file has carried since the iteration was **The question**, in the words this file has carried since the iteration was
@ -303,28 +364,7 @@ leg did.
Verified by feeding the gate a breaching run: `rss_ratio` 1.4, Verified by feeding the gate a breaching run: `rss_ratio` 1.4,
`overcap_vs_swap_x` 0.6 and `in_ram_cost_x` 12.0 are all rejected. `overcap_vs_swap_x` 0.6 and `in_ram_cost_x` 12.0 are all rejected.
- **Given** the `resident: all` read baseline, **when** re-measured, **then**
inside tolerance — no cost for a feature not used. ✅ Verified as a
by-product of the residency leg: `resident: all` is unchanged at 1 354 554
reads/sec uncapped, and every other db-bench leg still runs, which the
keys-resident classes had briefly broken by forcing `WO_DATA` module-wide.
- **Given** a `resident: keys` table larger than RAM, **when** read randomly,
**then** its read cost is measured against the resident baseline on its own
read path. ✅ Measured 2026-08-30 and the answer is qualified: **1.53×**
faster than letting the kernel swap under a cap that binds one and not the
other — real, but nowhere near iteration 1's 273× swap figure would suggest,
because cgroup limits charge the page cache, so moving rows to a file does
not escape a container memory limit. The unambiguous win is footprint:
**2.55×** smaller resident set. Full method, numbers and the failed first
attempt above; gated by `residency.*`.
Outstanding:
- **Given** `durable: true` and no `WO_DATA`, **when** the program starts,
**then** it refuses. *(task 6 — today this combination silently discards
every write)*
- **Given** the resident footprint crossing the budget, **when** it does,
**then** a refusal naming the table and the annotation. *(task 6)*
## Out Of Scope ## Out Of Scope
- **Checkpoint and compaction** — [3](03-wal-checkpoint.md). Boot rebuilds the - **Checkpoint and compaction** — [3](03-wal-checkpoint.md). Boot rebuilds the
@ -362,7 +402,67 @@ Outstanding:
5. **The budget is bytes, not rows** — a text-heavy row and an Int-only row 5. **The budget is bytes, not rows** — a text-heavy row and an Int-only row
differ by 3.3× (measured, databasev2 1), so a row count cannot bound RAM. differ by 3.3× (measured, databasev2 1), so a row count cannot bound RAM.
## History — two corrections worth keeping **Task 6a — forks 1–6 locked 2026-09-09, fork 7 added 2026-09-10** (auto-approved
for autonomous execution; the frontmatter's `review_pending` asks for the
developer's second review before close):
1. **The escape hatch is the environment sentinel `WO_EPHEMERAL=1`, exact
value.** Not `WO_DATA=:memory:` — `WO_DATA` stays a path and only a path,
which also keeps [7](07-single-file-db.md)'s file-vs-directory parse free
of sentinels — and no fixture edits. `@table(durable: false)` remains the
per-table declaration; the sentinel is the whole-program one.
2. **Sentinel rules.** Honoured only when `WO_DATA` is unset or empty.
`WO_EPHEMERAL` set together with `WO_DATA` is a startup refusal, exit 2,
naming the conflict. Any value other than `1` is the same refusal. A
`resident: keys` class still refuses through the existing loop — the
sentinel does not bypass it. One stderr boot notice when the sentinel is
in force.
3. **The refusal contract** for `durable: true` (the default) with no
`WO_DATA`: exit 2, one stderr line naming the first default-durable class
and all three ways forward — `WO_DATA=<dir>`, `WO_EPHEMERAL=1`,
`@table(durable: false)`. A second loop in `runtime/src/main.c`'s existing
startup-refusal block, beside the `resident: keys` one. No "+N more".
4. **Startup-only.** `db.c`'s guards are untouched, so the ephemeral path is
byte for byte today's RAM path; nothing on the data path learns a flag.
5. **The byte budget (6b) leaves this iteration** for
[5](05-bounded-tables-eviction.md), as a new Phase A — a per-table
resident byte counter feeding its pressure signal. 5 stays
`readiness: refine`; the design inputs go there as notes for its own
brainstorm, and the spec's three budget obligations become 5's acceptance
criteria.
6. **Library-owned durable tables bind every consumer.**
[porch's store](../../examples/porch/middleware/store.wo) declares its
tables with the default, so any program that `use`s it needs `WO_DATA` or
`WO_EPHEMERAL=1` as a whole-program requirement. No consumer-side
override: the library author's declaration is the declaration. Settled.
7. **The table bit in the `.wob` (v8), so durability rules apply to `@table`
classes only** — added 2026-09-10 after the first cut refused every
class-bearing program (fibers' `Tick`, subprocess's `ConnMsg`, log-watcher's
`CronEntry` are plain classes, and v7 spelled `durable: true` as the mere
absence of the volatile bit). `WO_CLASSF_TABLE` 0x08 is set from the
emitter's `cr_is_table`; the loader refuses the two storage bits without
it; `main.c`'s two refusal loops skip classes without it; a program with
no durable table does not consult `WO_EPHEMERAL` at all (the
`WO_DATA`+`WO_EPHEMERAL` conflict still refuses regardless). A v7 image is
refused by the version check, as v6 was by v7. The alternative — teaching
the runtime to infer "table" from the presence of indexes or an `insert`
site — was rejected: a fact the compiler already holds belongs in the
image, not re-derived.
## History — three corrections worth keeping
**Task 6a's first cut refused every class-bearing program (2026-09-09→10).**
The refusal keyed on "flags lack `WO_CLASSF_VOLATILE`", and the v7 image had
no bit saying "this class is a `@table`" — so `class Tick` in fibers looked
exactly like a default-durable table, and gates that never touch a table
(fibers, subprocess, log-watcher) had to export `WO_EPHEMERAL=1` to start.
Corrected the next day by recording the missing fact in the image (`.wob` v8,
`WO_CLASSF_TABLE`, fork 7), then re-measuring every gate without its export
and keeping the sentinel only where the program really refused: the corpus,
db-bench and db-actor (their own tables), chat (porch's store, fork 6) and
wmux's client legs (the server image, no `WO_DATA`). The lesson: "does this
gate run a table program" is answered by running it, not by reading the
example's own source — a `use`d library's declaration counts.
**The three-mode design was replaced.** Earlier drafts had **The three-mode design was replaced.** Earlier drafts had
`mode: ram | durable | cold`. `cold` conflated two independent properties and `mode: ram | durable | cold`. `cold` conflated two independent properties and
@ -381,7 +481,11 @@ fine, the plan's storage steps still read as plumbing. Measured instead:
`wo_row_ptr` returns a `db_row *` into a slab and has 11 call sites, `table.c` `wo_row_ptr` returns a `db_row *` into a slab and has 11 call sites, `table.c`
has 37 slab references, `db.c:105-181` walks slabs for scans, `enc_val` has 37 slab references, `db.c:105-181` walks slabs for scans, `enc_val`
serialises *from* the slab, and **no operation exists that drops a row's payload serialises *from* the slab, and **no operation exists that drops a row's payload
while keeping its index entries**. Hence the 5a–5d split. 5c and 5d need their while keeping its index entries**. Hence the 5a–5d split. Both 5c questions
own write-ups, and the two open design questions for 5c are whether the id hash are settled by what landed (2026-08-29): the id hash stores the LOG OFFSET in
stores offsets in place of slot indices or gains a parallel map, and what the place of the slot index — one map, no parallel one — which is also why the
new operation does about the unique shadows, which currently point at slots. `delete` corruption above was possible, since a reader that trusts that value
as a slot indexes a slab with a byte offset; and the unique shadows keep their
bucket candidates and resolve them through the same borrow, one `pread` per
candidate, never a slot dereference. The write-ups are the 5c/5d acceptance
bullets above and the 2026-08-29/30 board entries.

View file

@ -385,25 +385,36 @@ plus wherever per-table accounting lands from Task 6.
so a program that declares durability and is given nowhere to put it silently so a program that declares durability and is given nowhere to put it silently
loses everything. Refuse at startup, naming the first durable class found. loses everything. Refuse at startup, naming the first durable class found.
This is the single most valuable line in the plan and is independent of This is the single most valuable line in the plan and is independent of
residency. residency. *Decided 2026-09-09: exit 2, one stderr line, the first
default-durable class and all three ways forward; a second loop in `main.c`'s
existing startup-refusal block.*
- [ ] Provide the escape hatch the refusal implies: a program that genuinely - [ ] Provide the escape hatch the refusal implies: a program that genuinely
wants an ephemeral run must be able to say so, either by declaring its tables wants an ephemeral run must be able to say so, either by declaring its tables
volatile or by an explicit opt-out flag. Decide which and document it — a volatile or by an explicit opt-out flag. Decide which and document it — a
refusal with no stated way forward is a worse bug than the silent loss. refusal with no stated way forward is a worse bug than the silent loss.
*Decided 2026-09-09: `WO_EPHEMERAL=1`, exact value, honoured only when
`WO_DATA` is unset or empty; set together is a refusal; any other value is a
refusal; `@table(durable: false)` stays the per-table form.*
- [ ] Implement the byte budget: estimated resident footprint across all - [ ] Implement the byte budget: estimated resident footprint across all
tables, breached loudly with a message naming the largest offending table and tables, breached loudly with a message naming the largest offending table and
the exact annotation to add. the exact annotation to add. *Moved to databasev2 5 Phase A (2026-09-09).*
- [ ] Default the budget to a fraction of host-detected available memory, **not - [ ] Default the budget to a fraction of host-detected available memory, **not
to "none"** — a budget nobody sets cannot produce the diagnostic that is this to "none"** — a budget nobody sets cannot produce the diagnostic that is this
design's main deliverable, and the 120 GB developer would still meet the OOM design's main deliverable, and the 120 GB developer would still meet the OOM
killer. Use a conservative placeholder fraction and mark the value explicitly killer. Use a conservative placeholder fraction and mark the value explicitly
unset-pending in both the code comment and the iteration: the real number unset-pending in both the code comment and the iteration: the real number
comes from databasev2 1's swap-onset measurement. comes from databasev2 1's swap-onset measurement. *Moved to databasev2 5
(2026-09-09) as a note for its brainstorm: bytes XOR fraction,
`MemAvailable` and the cgroup v2 limits as the denominator, an itemised
measured reserve with a floor. Databasev2 1 found there is no swap onset to
derive from.*
- [ ] Make the accounting's error bound explicit where it is documented. It - [ ] Make the accounting's error bound explicit where it is documented. It
estimates RSS; it is not RSS, and pretending otherwise would make the budget estimates RSS; it is not RSS, and pretending otherwise would make the budget
untrustworthy the first time someone checked it. untrustworthy the first time someone checked it. *Moved to databasev2 5
(2026-09-09) — an acceptance criterion there.*
- [ ] Add CLI-smoke coverage for both refusals, including the exit code and the - [ ] Add CLI-smoke coverage for both refusals, including the exit code and the
first line of stderr — the shape scripts depend on. first line of stderr — the shape scripts depend on. *2026-09-09: one refusal
here now (6a); the budget's is 5's.*
- [ ] Verify: `bash runtime/test/cli_smoke.sh`, `just wovm-test`, - [ ] Verify: `bash runtime/test/cli_smoke.sh`, `just wovm-test`,
`just employee`, `just db-actor` green; every sample that sets `WO_DATA` `just employee`, `just db-actor` green; every sample that sets `WO_DATA`
still runs, and one that does not is now refused or explicitly opted out. still runs, and one that does not is now refused or explicitly opted out.

View file

@ -215,8 +215,14 @@ as the code — not afterwards:
combination silently loses everything, which is the worst failure mode in the combination silently loses everything, which is the worst failure mode in the
current engine. A program that declares durability and is given nowhere to put current engine. A program that declares durability and is given nowhere to put
it must not start. This is independent of residency and is arguably the most it must not start. This is independent of residency and is arguably the most
valuable single line in the spec. valuable single line in the spec. *Decided 2026-09-09 (databasev2 2 task 6a):
- **A byte budget that exists by default, breached loudly.** The budget bounds exit 2, one stderr line naming the first default-durable class and the three
ways forward — `WO_DATA=<dir>`, `WO_EPHEMERAL=1`, `@table(durable: false)`;
the whole-program escape hatch is `WO_EPHEMERAL=1`, exact value, honoured
only when `WO_DATA` is unset.*
- **A byte budget that exists by default, breached loudly.** *Moved to
[databasev2 5](../../stories/databasev2/05-bounded-tables-eviction.md) Phase A
on 2026-09-09; the text below stands as the original obligation.* The budget bounds
estimated resident footprint across all tables. On breach the program refuses estimated resident footprint across all tables. On breach the program refuses
with a message naming the largest offending table and the exact annotation to with a message naming the largest offending table and the exact annotation to
add, so the 120 GB developer meets a diagnostic at 32 GB rather than the OOM add, so the 120 GB developer meets a diagnostic at 32 GB rather than the OOM
@ -298,7 +304,9 @@ Acceptance is the story's Given/When/Then list; this is how each is exercised.
- **Compiler refusals** — a `compile-fail` fixture per diagnostic, each pinning - **Compiler refusals** — a `compile-fail` fixture per diagnostic, each pinning
the exact code. the exact code.
- **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup; - **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup;
a budget breach names the table and the annotation. a budget breach names the table and the annotation. *(2026-09-09: the first
is task 6a, escape hatch `WO_EPHEMERAL=1`; the budget breach moved to
databasev2 5.)*
- **Crash safety** — `kill -9` mid-append and mid-checkpoint on a - **Crash safety** — `kill -9` mid-append and mid-checkpoint on a
`resident: keys` table; replay loses no acked write and no row appears twice. `resident: keys` table; replay loses no acked write and no row appears twice.
- **Performance** — new baseline rows for the `resident: keys` read path with - **Performance** — new baseline rows for the `resident: keys` read path with
@ -337,10 +345,13 @@ unchanged.
loud one, and both listed here so neither arrives as a surprise: loud one, and both listed here so neither arrives as a surprise:
1. `durable: true` (the default) with no `WO_DATA` becomes a startup refusal. 1. `durable: true` (the default) with no `WO_DATA` becomes a startup refusal.
Today it silently discards every write. Today it silently discards every write. *(Decided 2026-09-09:
`WO_EPHEMERAL=1` opts the whole program out.)*
2. Total estimated resident footprint crossing the default budget fraction 2. Total estimated resident footprint crossing the default budget fraction
becomes a startup or insert refusal. Today the program slides into swap with becomes a startup or insert refusal. Today the program slides into swap with
no signal and is eventually killed. no signal and is eventually killed. *(Moved to databasev2 5 on 2026-09-09 —
no longer part of iteration 2's behaviour change.)*
Both are opt-out-able by explicit declaration. Neither is a data-format change, Both are opt-out-able by explicit declaration *(for the first: per table by
`@table(durable: false)`, or for the program by `WO_EPHEMERAL=1`)*. Neither is a data-format change,
so a rollback is a binary swap with no migration. so a rollback is a binary swap with no migration.