writeonce/docs/superpowers/specs/2026-08-26-table-residency-design.md

# databasev2 2 — table residency: the log as row store (for review)

> **Status:** design approved in brainstorm 2026-08-26, pending review of this
> document. Implements [databasev2 2](../../stories/databasev2/02-table-storage-modes.md),
> which is rewritten to match once this is approved. Amends
> [principle 7](../../00-principles.md) — already applied.
>
> Driving requirement, from the developer: **a 120 GB order table on a 32 GB
> host.** Not a tuning problem; no eviction policy fixes it. Small data-driven
> applications are well served by today's resident default and must not regress.

## Decisions taken (the brainstorm's forks, settled)

| Fork | Decision |
| --- | --- |
| One enum or two keys | **Two keys.** `durable:` and `resident:` answer two different developer questions ("do I need this after a restart?", "does it fit in RAM?"). One enum forces a name for each *combination*, which is what made a third value unreadable. |
| Mode vocabulary | **`resident: all \| keys`** and **`durable: true \| false`**. No `cold`, `tiered`, `paged`, `mmap` or `buffer` in the grammar. |
| Optional or mandatory | **Optional, both default to today's behaviour** (`durable: true`, `resident: all`). All 28 existing declarations compile unchanged; no goldens reblessed. |
| Which storage architecture | **One engine, log-structured.** The WAL already holds every row; keep an in-RAM id→offset map and read rows back with `pread`. No second engine. |
| Row cache | **None in user space.** The kernel page cache is the hot copy — the repo's own stated position in `exploration/postgresql/buffer-and-checkpoint.md`: "`pread` against an fd that already has its page cached is a memcpy… the page cache is the one cache we want", and the reason the engine avoids `O_DIRECT`. |
| `@unique` on a non-resident table | **Allowed; its index is unconditionally resident.** Settled here rather than deferred — see Constraints. |
| Budget unit | **Bytes** (estimated resident footprint). Rows is the meaningless unit: a text-heavy row and an Int-only row differ by 3.3× (measured, databasev2 1), so a row count cannot bound RAM. |
| Rejected architectures | `mmap` and a buffer pool stay out — see Alternatives rejected. `discarded.md`'s paged-engine rejection is amended to *partly revisited*, not reversed. |

## The problem, read off the engine

Facts, each verified in source rather than assumed:

- Rows live in `malloc`'d slabs of `DB_SLAB_ROWS` (256), allocated as a table
  grows and freed only at table teardown. The free-slot list recycles removed
  slots, so a delete-heavy table plateaus; a growing table only grows.
- **The ceiling is process RSS and nothing declares it.** `WO_HEAP_MB`
  (default 64 MiB) bounds the VM object arena; table storage is separate
  `malloc`. No knob says "this database may use at most N".
- No eviction, spill, paging or LRU exists anywhere in `database/src`.
- **Durability is process-global.** `main.c` opens one `shard-0.wal` when
  `WO_DATA` is set. `db.c` guards every WAL append with a null check on
  `vm->rt.wal`, so with no `WO_DATA` **every table is silently volatile** — a
  program can declare nothing and lose everything.
- An allocation failure is clean *in principle*: every `malloc` in the row
  encoder is checked and `DB_ERR_OOM` maps to `WO_T_OOM`. **Corrected 2026-08-27
  by measurement (databasev2 1): that path does not fire in practice.** With
  `vm.overcommit_memory = 0`, `malloc` succeeds and the process is SIGKILLed
  when it touches the pages — measured rc=137 at 360 000 rows under a 64 MiB
  cgroup cap. The checked-trap path belongs to the VM arena (`WO_HEAP_MB`,
  verified `trap 4 ... out of memory`), not to table storage, which has no
  ceiling at all. This makes the byte budget below the ONLY mechanism by which
  table storage can acquire one.

The measurements that bound the design, from iteration 22: durable inserts
≈4.5k/s against RAM ≈297k/s (the 66× fsync gap); reads 1.3M ops/s at p50 1µs
after the index probe landed. The read number is what a resident table buys and
what must not regress for tables that keep it.

### Why the arithmetic works

A 120 GB table at ~500 B/row is ≈240M rows. An id→offset entry is 16 bytes, so
the resident index is **≈3.8 GB** — comfortable inside 32 GB with room for two
secondary indexes of similar cost. That ratio is the whole design: **indexes
stay resident, rows do not.** It buys roughly two orders of magnitude of table
size, not infinity, and the spec says so plainly because a design sold as
unlimited gets deployed as if it were.

**What the trade costs, bounded by measurement (databasev2 1, 2026-08-27).**
Buying table size with disk reads is not free, and the price is large: random
reads across a table larger than RAM measured **273× slower** than resident ones
(1 851 166 reads/s against 6 771; p99 1 µs against 487 µs), and the transition is
a **step, not a curve** — there is no gentle region to operate in. That figure is
an *upper bound on the mechanism this spec does not use*: it is demand-paging of
anonymous memory through swap, 4 KiB per fault with no readahead, whereas
`resident: keys` `pread`s from the WAL through the page cache, which gets
readahead and a shared cache. The constant should therefore be better — **but
that is a hypothesis and task 7 must measure it, not inherit it.** Two things
follow regardless: `resident: keys` must stay opt-in per table (it is), and the
hot-set question is not deferrable decoration — it is
[iteration 5](../../stories/databasev2/05-bounded-tables-eviction.md) and it
decides whether this design is usable for anything read-heavy.

## The design

### Grammar

Two new `@table` arguments, both optional. The parser's argument loop already
matches `name` and `index` and rejects anything else with a catalogued
diagnostic; these are two more arms and an updated message. `Ast.table_cfg`
grows two fields beside `table_name` and `indexes`.

| Argument | Values | Default | Meaning |
| --- | --- | --- | --- |
| `durable` | `true`, `false` | `true` | `false` skips the WAL append entirely: no record, no fsync, ack from RAM, table empty after restart. |
| `resident` | `all`, `keys` | `all` | `keys` keeps the id map, every secondary index and every unique shadow in RAM; rows are read from the log by offset. |

`true`/`false` are already keyword tokens; `all`/`keys` are parsed as the same
bare identifiers the `index:` argument's column list already accepts. No lexer
change.

**Named `keys`, not `index`, on review (2026-08-26).** An earlier draft used
the value `index`, which put `index` in value position while `index:` is also a
key — `@table(index: [customer], resident: index)` read awkwardly (that line is
the rejected spelling, quoted). `keys` also
puts both values on one axis: `all` and `keys` each answer "what row data stays
resident", where `all`/`index` mixed a quantity with a structure name. Neither
`all` nor `keys` is a keyword or a builtin (`key_at`/`val_at` exist; bare `keys`
does not). `resident: none` was considered and rejected as overclaiming — the
indexes are very much resident.

### The four combinations

| `durable` | `resident` | Status |
| --- | --- | --- |
| `true` | `all` | Today's behaviour. The default. Reads at memory speed. |
| `false` | `all` | Volatile scratch: sessions, rate-limit counters, idempotency keys. Skips the 66× fsync cost. What porch 1–3 need. |
| `true` | `keys` | The 120 GB case. Rows in the log, indexes resident, `pread` on read. |
| `false` | `keys` | **Refused at compile time.** Rows would have nowhere to be read from. |

### Read, write and recovery paths

- **Insert** — unchanged for `resident: all`. For `resident: keys`: encode and
  append the record as today, then record id→offset in the resident map instead
  of retaining the row in a slab. The WAL append is already the durable write;
  this stops discarding its payload.
- **Read by id** — resident map lookup, then `scan_record` at the offset (which
  already `pread`s and verifies the CRC), skip the record header, and `dec_val`
  each field. Both functions already exist in `wal.c` and are already exercised
  by replay; the change is that they are called on demand rather than only at
  boot.
- **Update** — ~~append a new record, repoint the offset~~. **SUPERSEDED
  2026-08-30** by
  [`2026-08-30-keys-resident-delta-updates-design.md`](2026-08-30-keys-resident-delta-updates-design.md).
  This line was written before any of the mode was built and describes a
  full-row append, which the motivating workload rejects: a product catalogue
  changes one narrow field of a wide row on every order, so a full-row append
  rewrites every field to move one integer. Updates append a **delta** carrying
  a back-pointer, folded on read, with compaction doing the fold that keeps
  chains short. The superseded-record-becomes-garbage half is still true.
- **Delete** — append a tombstone, drop the id from the map and every index.
- **Scan** — a sequential walk of the log, which is the case log-structured
  storage is best at. Cost changes from memory-speed to sequential-disk; the
  query surface is unchanged in *meaning* and materially different in *cost*.
- **Boot** — replay rebuilds the offset map by scanning the log. Correct, and
  O(entire history), which is the honest cost of shipping this before
  [databasev2 3](../../stories/databasev2/03-wal-checkpoint.md).

### Row encoding: nothing to build — corrected 2026-08-26

**An earlier draft of this section was wrong and claimed the opposite.** It said
the on-disk record was pointer-bearing and that re-encoding it was "the one real
rewrite" and the substantive engineering of this iteration. That came from
reading `table.c`'s `db_val_encode`, which builds the **in-memory slot**, and
inferring the file format from it. The file format is a *separate* encoding in
`wal.c`, and it has been flat since iteration 9.

What is already there, verified:

- `wal.c`'s `enc_val` inlines every kind recursively with no pointer anywhere —
  text and bytes as length-then-bytes, owned as class id then fields, multi as
  element kind, length, items, map as key kind, value kind, length, pairs.
  GCREF is never stored and never logged.
- `dec_val` reads that back and allocates fresh engine-owned values.
- A record is `WO_WAL_INSERT | class_id | id | <value per field>`, wrapped in
  the `len|crc|payload|mark` frame.
- `scan_record(fd, off, …)` already `pread`s the record at an arbitrary offset
  and verifies its CRC.

So the record is already position-independent, already carries the class id and
row id, and is already randomly addressable. The in-memory slot representation
needs **no change at all**, because it was never what reached the file.

**Where the real work is instead: capturing the offset.**
`wo_wal_append_insert` calls `stage()` into a buffer (opened at 1 MiB in
`main.c`), so a record's final file offset is not known at append time — only
when that buffer flushes. Threading an accurate offset back to the caller
through a buffered writer, and keeping it correct across a partial flush and a
torn tail, is the delicate piece of this iteration. It is a much better-defined
problem than the rewrite this section used to describe, and it is bounded to
`wal.c`'s staging path plus the map that consumes it.

Consequence for the plan: this iteration is cheaper and lower-risk than first
estimated. The task that was to perform the rewrite is deleted rather than
reduced.

### Constraints across the residency boundary

- **`@unique` is allowed, and its index is unconditionally resident.** A shadow
  check cannot scan slabs that are not there, so correctness requires the unique
  column's index in RAM. Cost is one hash entry per row — the same order as the
  id map, so a unique column roughly doubles the resident index. Stated at the
  declaration so the developer can see what they bought. Silently checking only
  resident rows is the one outcome that must never ship.
- **Foreign-key restrict works unchanged.** It is a secondary-index probe, and
  secondary indexes are resident.
- **`ref` navigation works unchanged**, at the cost of a `pread` per hop.
- **A `resident: all` table may hold a `ref` into a `resident: keys` table**
  and vice versa — both are durable, so neither evaporates. This is the case
  that a `durable: false` table genuinely breaks, below.

### What the compiler enforces

Four refusals, each a catalogued `WO-E1xx` diagnostic added in the same change
as the code — not afterwards:

1. `durable: false` with `resident: keys` — the meaningless combination.
2. An unknown value for either argument, or either argument given twice.
3. **A `durable: true` table holding a `ref` into a `durable: false` table.**
   A persistent row cannot reference one that evaporates on restart; FK restrict
   cannot save it and the dangling reference is provable from the class table.
   This is the highest-value check in the iteration.
4. A `cold`, `tiered`, `ram` or other retired mode word — refused with a message
   naming the two real arguments, so the vocabulary explored during the
   brainstorm does not become folklore.

### What the runtime enforces

- **`durable: true` with no `WO_DATA` is a startup refusal.** Today this
  combination silently loses everything, which is the worst failure mode in the
  current engine. A program that declares durability and is given nowhere to put
  it must not start. This is independent of residency and is arguably the most
  valuable single line in the spec.
- **A byte budget that exists by default, breached loudly.** The budget bounds
  estimated resident footprint across all tables. On breach the program refuses
  with a message naming the largest offending table and the exact annotation to
  add, so the 120 GB developer meets a diagnostic at 32 GB rather than the OOM
  killer.

  **The default must not be "no budget"** — that was a contradiction in the
  first draft of this spec: a budget nobody sets cannot produce the diagnostic
  that is this design's main deliverable, and the ERP developer would still meet
  the OOM killer. So the default is a **fraction of host-detected available
  memory**, overridable by an environment variable and by a per-program
  declaration. Choosing that fraction is the one number this spec cannot supply:
  it comes from [databasev2 1](../../stories/databasev2/01-ram-ceiling-measurement.md)'s
  swap-onset measurement, which is why 1 sequences before 2. Until 1 lands,
  implement the mechanism with a conservative placeholder fraction and treat the
  value as unset rather than settled.

  Consequence to accept deliberately: a program that today grows past that
  fraction and survives on a large host will now refuse. That is the intended
  behaviour change — it converts an invisible slide into swap into a startup
  error — but it *is* a behaviour change, and it is the second of the two
  breaks listed under Migration.
- `WO_DATA` remains the data-directory location. It stops being the durability
  switch; the declaration is.

### Format

The class descriptor carries both properties, so the runtime never re-derives
them. This moves `WOB_VERSION` (currently 6) and the format contract in the same
commit as the code. An older image is refused on version rather than misread.

## Why this shape

- **It keeps one engine and one source of truth.** The log *is* the database.
  Nothing here adds a second storage system, which is what the 2026-08-18
  rejection was actually about.
- **It costs nothing for tables that do not use it.** A `resident: all` table
  takes the same path it takes today. The 1.3M ops/s read baseline is the
  regression gate, and a measurable regression there is grounds to reject the
  implementation rather than tune it.
- **The failure mode becomes a diagnostic.** Both exits **as measured** in
  databasev2 1 (2026-08-27) are replaced by a refusal that names the fix — and
  the measurement made this argument stronger than the draft that named "swap
  thrash and the OOM killer". What actually happens is **SIGKILL signal 9** with
  swap off (no trap: overcommit lets `malloc` succeed, the kernel kills on page
  touch, so the checked path never runs) or **exit 0 while serving from disk**
  with swap on. Neither is a diagnostic; one is silence and the other is a
  corpse. A declared budget is the only way this engine can say anything at all
  before either.
- **It is declared, not automatic.** No threshold heuristic, no performance
  cliff the compiler cannot explain. Consistent with a language whose thesis is
  that the compiler tells you the truth.

## Alternatives rejected

| Alternative | Why not |
| --- | --- |
| **`mmap` the row file** | Gives up precise ack-after-fsync for the kernel's flush schedule, which is the one guarantee this design will not trade. The repo's own mmap study only ever proposed it read-only for segment lookups. Since records are *already* flat and position-independent, mmap remains available later as a pure read-path optimisation over the same file — recorded, not adopted. |
| **Buffer pool with dirty-page tracking** | This is the Rust-era phase-12 design in `exploration/postgresql/buffer-and-checkpoint.md` (`CachedRow { bytes, dirty }`, `WO_CACHE_ROWS` LRU) that died with that track. It duplicates the kernel page cache, and the page cache is explicitly the cache this project wants. |
| **Paged B-tree engine** | Rejected 2026-08-18 and still rejected. Reading rows from the log we already write is not this. |
| **A three-valued `mode:` enum** | Needs a name per combination. The brainstorm demonstrated that the third name is unwriteable before its mechanism is decided. |
| **Automatic spill at a threshold** | No annotation needed, but unpredictable cliffs and magic the compiler cannot explain. |
| **Disk-backed by default** | Costs every application the 1µs read, including the small ones explicitly said to be well served today. |

## Proof plan

Acceptance is the story's Given/When/Then list; this is how each is exercised.

- **Compatibility** — all 28 existing `@table` declarations compile untouched;
  `just employee`, `just db-actor`, `just web-app`, `just site`, `oop-accept`
  unchanged; no golden reblessed.
- **Volatility** — a `durable: false` table produces no WAL growth (measured,
  not asserted) and is empty after restart while durable siblings replay intact.
- **Residency correctness** — a `resident: keys` table larger than the
  configured budget returns every row correctly by id, byte-identical including
  every heap-valued column, and scans in full.
- **Constraints across the boundary** — `@unique` refuses a duplicate whose
  conflicting row is not resident; FK restrict refuses a delete whose only
  referrer is not resident. Corpus fixtures, both.
- **Compiler refusals** — a `compile-fail` fixture per diagnostic, each pinning
  the exact code.
- **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup;
  a budget breach names the table and the annotation.
- **Crash safety** — `kill -9` mid-append and mid-checkpoint on a
  `resident: keys` table; replay loses no acked write and no row appears twice.
- **Performance** — new baseline rows for the `resident: keys` read path with
  its amplification versus resident, published in `perf-targets.md` as a number
  a developer can plan around; and a regression check that resident tables did
  not move.
- **Sanitisers** — ASan on the new decode path, which is where the bugs are.

## Out of scope

- **Checkpoint and compaction** — [databasev2 3](../../stories/databasev2/03-wal-checkpoint.md).
  This spec's boot cost is O(history) until 3 lands, and 3's snapshot should
  persist the offset map so boot stops rescanning. That coupling is stated in
  both documents.
- **Eviction policy and a resident row cache** — [databasev2 5](../../stories/databasev2/05-bounded-tables-eviction.md).
  This design needs neither: rows are either all resident or none are.
- **io_uring on the read path** — a real question that only exists after this
  lands; noted in [databasev2 4](../../stories/databasev2/04-io-uring-commit.md),
  not folded into it.
- **Per-shard residency** — the owner shard owns the store and the log. A
  `durable: false` table arguably need not live on the owner at all, which would
  be dramatically faster and a different consistency story. Recorded as a
  candidate, deliberately not decided here.
- **Migrating an existing dataset between settings**, and a WAL written when a
  table had different settings: refuse clearly on mismatch, do not convert.
- **`transaction { }` and `@table` feature flags** — language iteration 18,
  approved spec, left whole.

## Migration

Nothing to convert. Both arguments default to present behaviour, so every
existing program, manifest, corpus fixture and golden compiles and runs
unchanged.

**Two deliberate behaviour changes**, both converting a silent failure into a
loud one, and both listed here so neither arrives as a surprise:

1. `durable: true` (the default) with no `WO_DATA` becomes a startup refusal.
   Today it silently discards every write.
2. Total estimated resident footprint crossing the default budget fraction
   becomes a startup or insert refusal. Today the program slides into swap with
   no signal and is eventually killed.

Both are opt-out-able by explicit declaration. Neither is a data-format change,
so a rollback is a binary swap with no migration.