writeonce/docs/superpowers/specs/2026-08-26-table-residency-design.md

databasev2 2 — table residency: the log as row store (for review)

Status: design approved in brainstorm 2026-08-26, pending review of this document. Implements databasev2 2, which is rewritten to match once this is approved. Amends principle 7 — already applied.

Driving requirement, from the developer: a 120 GB order table on a 32 GB host. Not a tuning problem; no eviction policy fixes it. Small data-driven applications are well served by today's resident default and must not regress.

Decisions taken (the brainstorm's forks, settled)

Fork	Decision
One enum or two keys	Two keys. durable: and resident: answer two different developer questions ("do I need this after a restart?", "does it fit in RAM?"). One enum forces a name for each combination, which is what made a third value unreadable.
Mode vocabulary	resident: all | keys and durable: true | false. No cold, tiered, paged, mmap or buffer in the grammar.
Optional or mandatory	Optional, both default to today's behaviour (durable: true, resident: all). All 28 existing declarations compile unchanged; no goldens reblessed.
Which storage architecture	One engine, log-structured. The WAL already holds every row; keep an in-RAM id→offset map and read rows back with pread. No second engine.
Row cache	None in user space. The kernel page cache is the hot copy — the repo's own stated position in exploration/postgresql/buffer-and-checkpoint.md: "pread against an fd that already has its page cached is a memcpy… the page cache is the one cache we want", and the reason the engine avoids O_DIRECT.
@unique on a non-resident table	Allowed; its index is unconditionally resident. Settled here rather than deferred — see Constraints.
Budget unit	Bytes (estimated resident footprint). Rows is the meaningless unit: a text-heavy row and an Int-only row differ by 3.3× (measured, databasev2 1), so a row count cannot bound RAM.
Rejected architectures	mmap and a buffer pool stay out — see Alternatives rejected. discarded.md's paged-engine rejection is amended to partly revisited, not reversed.

The problem, read off the engine

Facts, each verified in source rather than assumed:

• Rows live in malloc'd slabs of DB_SLAB_ROWS (256), allocated as a table grows and freed only at table teardown. The free-slot list recycles removed slots, so a delete-heavy table plateaus; a growing table only grows.
• The ceiling is process RSS and nothing declares it. WO_HEAP_MB (default 64 MiB) bounds the VM object arena; table storage is separate malloc. No knob says "this database may use at most N".
• No eviction, spill, paging or LRU exists anywhere in database/src.
• Durability is process-global. main.c opens one shard-0.wal when WO_DATA is set. db.c guards every WAL append with a null check on vm->rt.wal, so with no WO_DATA every table is silently volatile — a program can declare nothing and lose everything.
• An allocation failure is clean in principle: every malloc in the row encoder is checked and DB_ERR_OOM maps to WO_T_OOM. Corrected 2026-08-27 by measurement (databasev2 1): that path does not fire in practice. With vm.overcommit_memory = 0, malloc succeeds and the process is SIGKILLed when it touches the pages — measured rc=137 at 360 000 rows under a 64 MiB cgroup cap. The checked-trap path belongs to the VM arena (WO_HEAP_MB, verified trap 4 ... out of memory), not to table storage, which has no ceiling at all. This makes the byte budget below the ONLY mechanism by which table storage can acquire one.

The measurements that bound the design, from iteration 22: durable inserts ≈4.5k/s against RAM ≈297k/s (the 66× fsync gap); reads 1.3M ops/s at p50 1µs after the index probe landed. The read number is what a resident table buys and what must not regress for tables that keep it.

Why the arithmetic works

A 120 GB table at ~500 B/row is ≈240M rows. An id→offset entry is 16 bytes, so the resident index is ≈3.8 GB — comfortable inside 32 GB with room for two secondary indexes of similar cost. That ratio is the whole design: indexes stay resident, rows do not. It buys roughly two orders of magnitude of table size, not infinity, and the spec says so plainly because a design sold as unlimited gets deployed as if it were.

What the trade costs, bounded by measurement (databasev2 1, 2026-08-27). Buying table size with disk reads is not free, and the price is large: random reads across a table larger than RAM measured 273× slower than resident ones (1 851 166 reads/s against 6 771; p99 1 µs against 487 µs), and the transition is a step, not a curve — there is no gentle region to operate in. That figure is an upper bound on the mechanism this spec does not use: it is demand-paging of anonymous memory through swap, 4 KiB per fault with no readahead, whereas resident: keys preads from the WAL through the page cache, which gets readahead and a shared cache. The constant should therefore be better — but that is a hypothesis and task 7 must measure it, not inherit it. Two things follow regardless: resident: keys must stay opt-in per table (it is), and the hot-set question is not deferrable decoration — it is iteration 5 and it decides whether this design is usable for anything read-heavy.

The design

Grammar

Two new @table arguments, both optional. The parser's argument loop already matches name and index and rejects anything else with a catalogued diagnostic; these are two more arms and an updated message. Ast.table_cfg grows two fields beside table_name and indexes.

Argument	Values	Default	Meaning
durable	true, false	true	false skips the WAL append entirely: no record, no fsync, ack from RAM, table empty after restart.
resident	all, keys	all	keys keeps the id map, every secondary index and every unique shadow in RAM; rows are read from the log by offset.

true/false are already keyword tokens; all/keys are parsed as the same bare identifiers the index: argument's column list already accepts. No lexer change.

Named keys, not index, on review (2026-08-26). An earlier draft used the value index, which put index in value position while index: is also a key — @table(index: [customer], resident: index) read awkwardly (that line is the rejected spelling, quoted). keys also puts both values on one axis: all and keys each answer "what row data stays resident", where all/index mixed a quantity with a structure name. Neither all nor keys is a keyword or a builtin (key_at/val_at exist; bare keys does not). resident: none was considered and rejected as overclaiming — the indexes are very much resident.

The four combinations

durable	resident	Status
true	all	Today's behaviour. The default. Reads at memory speed.
false	all	Volatile scratch: sessions, rate-limit counters, idempotency keys. Skips the 66× fsync cost. What porch 1–3 need.
true	keys	The 120 GB case. Rows in the log, indexes resident, pread on read.
false	keys	Refused at compile time. Rows would have nowhere to be read from.

Read, write and recovery paths

• Insert — unchanged for resident: all. For resident: keys: encode and append the record as today, then record id→offset in the resident map instead of retaining the row in a slab. The WAL append is already the durable write; this stops discarding its payload.
• Read by id — resident map lookup, then scan_record at the offset (which already preads and verifies the CRC), skip the record header, and dec_val each field. Both functions already exist in wal.c and are already exercised by replay; the change is that they are called on demand rather than only at boot.
• Update — append a new record, repoint the offset. SUPERSEDED 2026-08-30 by 2026-08-30-keys-resident-delta-updates-design.md. This line was written before any of the mode was built and describes a full-row append, which the motivating workload rejects: a product catalogue changes one narrow field of a wide row on every order, so a full-row append rewrites every field to move one integer. Updates append a delta carrying a back-pointer, folded on read, with compaction doing the fold that keeps chains short. The superseded-record-becomes-garbage half is still true.
• Delete — append a tombstone, drop the id from the map and every index.
• Scan — a sequential walk of the log, which is the case log-structured storage is best at. Cost changes from memory-speed to sequential-disk; the query surface is unchanged in meaning and materially different in cost.
• Boot — replay rebuilds the offset map by scanning the log. Correct, and O(entire history), which is the honest cost of shipping this before databasev2 3.

Row encoding: nothing to build — corrected 2026-08-26

An earlier draft of this section was wrong and claimed the opposite. It said the on-disk record was pointer-bearing and that re-encoding it was "the one real rewrite" and the substantive engineering of this iteration. That came from reading table.c's db_val_encode, which builds the in-memory slot, and inferring the file format from it. The file format is a separate encoding in wal.c, and it has been flat since iteration 9.

What is already there, verified:

• wal.c's enc_val inlines every kind recursively with no pointer anywhere — text and bytes as length-then-bytes, owned as class id then fields, multi as element kind, length, items, map as key kind, value kind, length, pairs. GCREF is never stored and never logged.
• dec_val reads that back and allocates fresh engine-owned values.
• A record is WO_WAL_INSERT | class_id | id | <value per field>, wrapped in the len|crc|payload|mark frame.
• scan_record(fd, off, …) already preads the record at an arbitrary offset and verifies its CRC.

So the record is already position-independent, already carries the class id and row id, and is already randomly addressable. The in-memory slot representation needs no change at all, because it was never what reached the file.

Where the real work is instead: capturing the offset. wo_wal_append_insert calls stage() into a buffer (opened at 1 MiB in main.c), so a record's final file offset is not known at append time — only when that buffer flushes. Threading an accurate offset back to the caller through a buffered writer, and keeping it correct across a partial flush and a torn tail, is the delicate piece of this iteration. It is a much better-defined problem than the rewrite this section used to describe, and it is bounded to wal.c's staging path plus the map that consumes it.

Consequence for the plan: this iteration is cheaper and lower-risk than first estimated. The task that was to perform the rewrite is deleted rather than reduced.

Constraints across the residency boundary

• @unique is allowed, and its index is unconditionally resident. A shadow check cannot scan slabs that are not there, so correctness requires the unique column's index in RAM. Cost is one hash entry per row — the same order as the id map, so a unique column roughly doubles the resident index. Stated at the declaration so the developer can see what they bought. Silently checking only resident rows is the one outcome that must never ship.
• Foreign-key restrict works unchanged. It is a secondary-index probe, and secondary indexes are resident.
• ref navigation works unchanged, at the cost of a pread per hop.
• A resident: all table may hold a ref into a resident: keys table and vice versa — both are durable, so neither evaporates. This is the case that a durable: false table genuinely breaks, below.

What the compiler enforces

Four refusals, each a catalogued WO-E1xx diagnostic added in the same change as the code — not afterwards:

1. durable: false with resident: keys — the meaningless combination.
2. An unknown value for either argument, or either argument given twice.
3. A durable: true table holding a ref into a durable: false table. A persistent row cannot reference one that evaporates on restart; FK restrict cannot save it and the dangling reference is provable from the class table. This is the highest-value check in the iteration.
4. A cold, tiered, ram or other retired mode word — refused with a message naming the two real arguments, so the vocabulary explored during the brainstorm does not become folklore.

What the runtime enforces

• durable: true with no WO_DATA is a startup refusal. Today this combination silently loses everything, which is the worst failure mode in the current engine. A program that declares durability and is given nowhere to put it must not start. This is independent of residency and is arguably the most valuable single line in the spec.

• A byte budget that exists by default, breached loudly. The budget bounds estimated resident footprint across all tables. On breach the program refuses with a message naming the largest offending table and the exact annotation to add, so the 120 GB developer meets a diagnostic at 32 GB rather than the OOM killer.

  The default must not be "no budget" — that was a contradiction in the first draft of this spec: a budget nobody sets cannot produce the diagnostic that is this design's main deliverable, and the ERP developer would still meet the OOM killer. So the default is a fraction of host-detected available memory, overridable by an environment variable and by a per-program declaration. Choosing that fraction is the one number this spec cannot supply: it comes from databasev2 1's swap-onset measurement, which is why 1 sequences before 2. Until 1 lands, implement the mechanism with a conservative placeholder fraction and treat the value as unset rather than settled.

  Consequence to accept deliberately: a program that today grows past that fraction and survives on a large host will now refuse. That is the intended behaviour change — it converts an invisible slide into swap into a startup error — but it is a behaviour change, and it is the second of the two breaks listed under Migration.

• WO_DATA remains the data-directory location. It stops being the durability switch; the declaration is.

Format

The class descriptor carries both properties, so the runtime never re-derives them. This moves WOB_VERSION (currently 6) and the format contract in the same commit as the code. An older image is refused on version rather than misread.

Why this shape

• It keeps one engine and one source of truth. The log is the database. Nothing here adds a second storage system, which is what the 2026-08-18 rejection was actually about.
• It costs nothing for tables that do not use it. A resident: all table takes the same path it takes today. The 1.3M ops/s read baseline is the regression gate, and a measurable regression there is grounds to reject the implementation rather than tune it.
• The failure mode becomes a diagnostic. Both exits as measured in databasev2 1 (2026-08-27) are replaced by a refusal that names the fix — and the measurement made this argument stronger than the draft that named "swap thrash and the OOM killer". What actually happens is SIGKILL signal 9 with swap off (no trap: overcommit lets malloc succeed, the kernel kills on page touch, so the checked path never runs) or exit 0 while serving from disk with swap on. Neither is a diagnostic; one is silence and the other is a corpse. A declared budget is the only way this engine can say anything at all before either.
• It is declared, not automatic. No threshold heuristic, no performance cliff the compiler cannot explain. Consistent with a language whose thesis is that the compiler tells you the truth.

Alternatives rejected

Alternative	Why not
mmap the row file	Gives up precise ack-after-fsync for the kernel's flush schedule, which is the one guarantee this design will not trade. The repo's own mmap study only ever proposed it read-only for segment lookups. Since records are already flat and position-independent, mmap remains available later as a pure read-path optimisation over the same file — recorded, not adopted.
Buffer pool with dirty-page tracking	This is the Rust-era phase-12 design in exploration/postgresql/buffer-and-checkpoint.md (CachedRow { bytes, dirty }, WO_CACHE_ROWS LRU) that died with that track. It duplicates the kernel page cache, and the page cache is explicitly the cache this project wants.
Paged B-tree engine	Rejected 2026-08-18 and still rejected. Reading rows from the log we already write is not this.
A three-valued mode: enum	Needs a name per combination. The brainstorm demonstrated that the third name is unwriteable before its mechanism is decided.
Automatic spill at a threshold	No annotation needed, but unpredictable cliffs and magic the compiler cannot explain.
Disk-backed by default	Costs every application the 1µs read, including the small ones explicitly said to be well served today.

Proof plan

Acceptance is the story's Given/When/Then list; this is how each is exercised.

• Compatibility — all 28 existing @table declarations compile untouched; just employee, just db-actor, just web-app, just site, oop-accept unchanged; no golden reblessed.
• Volatility — a durable: false table produces no WAL growth (measured, not asserted) and is empty after restart while durable siblings replay intact.
• Residency correctness — a resident: keys table larger than the configured budget returns every row correctly by id, byte-identical including every heap-valued column, and scans in full.
• Constraints across the boundary — @unique refuses a duplicate whose conflicting row is not resident; FK restrict refuses a delete whose only referrer is not resident. Corpus fixtures, both.
• Compiler refusals — a compile-fail fixture per diagnostic, each pinning the exact code.
• Runtime refusals — durable: true with no WO_DATA fails at startup; a budget breach names the table and the annotation.
• Crash safety — kill -9 mid-append and mid-checkpoint on a resident: keys table; replay loses no acked write and no row appears twice.
• Performance — new baseline rows for the resident: keys read path with its amplification versus resident, published in perf-targets.md as a number a developer can plan around; and a regression check that resident tables did not move.
• Sanitisers — ASan on the new decode path, which is where the bugs are.

Out of scope

• Checkpoint and compaction — databasev2 3. This spec's boot cost is O(history) until 3 lands, and 3's snapshot should persist the offset map so boot stops rescanning. That coupling is stated in both documents.
• Eviction policy and a resident row cache — databasev2 5. This design needs neither: rows are either all resident or none are.
• io_uring on the read path — a real question that only exists after this lands; noted in databasev2 4, not folded into it.
• Per-shard residency — the owner shard owns the store and the log. A durable: false table arguably need not live on the owner at all, which would be dramatically faster and a different consistency story. Recorded as a candidate, deliberately not decided here.
• Migrating an existing dataset between settings, and a WAL written when a table had different settings: refuse clearly on mismatch, do not convert.
• transaction { } and @table feature flags — language iteration 18, approved spec, left whole.

Migration

Nothing to convert. Both arguments default to present behaviour, so every existing program, manifest, corpus fixture and golden compiles and runs unchanged.

Two deliberate behaviour changes, both converting a silent failure into a loud one, and both listed here so neither arrives as a surprise:

1. durable: true (the default) with no WO_DATA becomes a startup refusal. Today it silently discards every write.
2. Total estimated resident footprint crossing the default budget fraction becomes a startup or insert refusal. Today the program slides into swap with no signal and is eventually killed.

Both are opt-out-able by explicit declaration. Neither is a data-format change, so a rollback is a binary swap with no migration.