writeonce/docs/superpowers/plans/2026-08-26-table-residency.md

# databasev2 2 — table residency implementation plan

> **For agentic workers:** REQUIRED SUB-SKILL: use
> `superpowers:subagent-driven-development` (recommended) or
> `superpowers:executing-plans` to implement this plan task-by-task. Steps use
> checkbox (`- [ ]`) syntax for tracking.
>
> **This plan contains no code, deliberately.** That is a repo rule, not an
> omission: `docs/plan/discarded.md` records "Raw code in plan documents —
> plans carry concept, reason, and required behavior in words; the executor
> writes the code", and all six preceding plans in this directory have zero
> code fences. Every step below names the exact file and line region to change
> and the behaviour required; the executor writes the code and the tests.

**Goal:** let a table declare what it keeps in memory, so a 120 GB table works
on a 32 GB host, without changing anything for tables that fit.

**Architecture:** one storage engine, log-structured. The WAL already holds
every row; this plan stops discarding the payload — an in-RAM id→offset map
plus `pread` becomes the read path for tables that ask for it. Two optional
`@table` arguments, both defaulting to today's behaviour. No second engine, no
user-space row cache (the kernel page cache is the hot copy).

**Tech Stack:** OCaml (compiler front end, stdlib only), C11 (runtime and
engine, libc only), the existing `.wob` image format, `tests/corpus` +
`compiler/test/golden` for fixtures, `just` recipes as gates.

**Spec:** [`../specs/2026-08-26-table-residency-design.md`](../specs/2026-08-26-table-residency-design.md)

## Global constraints

- **Defaults preserve behaviour exactly.** `durable: true`, `resident: all`.
  All 28 existing `@table` declarations compile and run unchanged; no golden is
  reblessed. Any task that breaks this is wrong.
- **The resident read baseline must not regress.** 1.3M ops/s, p50 1µs
  (iteration 22). A measurable regression on `resident: all` tables is grounds
  to reject the implementation, not to tune it.
- **Durability semantics are untouched.** Ack after fsync, replay whole-or-
  nothing, torn tail dropped by CRC. `resident:` never changes whether an ack
  means durable.
- **Every new diagnostic is catalogued in the same commit** that adds it, in
  `docs/plan/oop-vm/01-error-catalog.md`. That catalog went stale once by doing
  this later; ten codes had to be back-filled on 2026-08-26.
- **C11 + libc only** in `runtime/` and `database/`; OCaml stdlib only in
  `compiler/`. No new dependency.
- **`WOB_VERSION` moves once**, in Task 3, with the format contract
  (`docs/plan/oop-vm/00-wob-format.md`) in the same commit.
- Gates that must be green at the end of every task: `just woc-test`,
  `just wovm-test`, `just oop-e2e`, `just employee`, `just db-actor`.
  Tasks 5 onward add `just db-bench`.

---

## Task 1 — grammar: the two arguments exist and default correctly

**Files:** modify `compiler/src/ast.ml` (`table_cfg` at :521-524),
`compiler/src/parser.ml` (`parse_table_cfg` at :302-345, `table_code` at :128),
`compiler/src/dump.ml` (the `@table` dump line);
create goldens under `compiler/test/golden/ast/`.

**Interfaces:**
- Produces: `Ast.table_cfg` gains two fields — a durability flag defaulting to
  true, and a residency selector defaulting to "all". Tasks 2, 3 and 4 read
  both off `class_info.table` (`compiler/src/types.ml:49`).

- [ ] Extend `table_cfg` at `ast.ml:521-524` with the two fields beside
  `table_name` and `indexes`. Both carry their default in the record's initial
  value so an unannotated `@table` is byte-identical to today.
- [ ] Add a `durable` arm to `parse_table_cfg`'s key match (`parser.ml:313`,
  beside `name` and `index`). Accept only the `true`/`false` keyword tokens —
  they already exist as `KwTrue`/`KwFalse`. Reject a second occurrence with
  `table_code`, matching the shape of the existing "given twice" failure at
  `:315`.
- [ ] Add a `resident` arm accepting the bare identifiers `all` and `keys`,
  read the same way the `index:` arm reads column identifiers (`:322-327`).
  Reject a second occurrence, and reject any other identifier.
- [ ] Update the unknown-argument message at `:334` to list all four supported
  keys. Add an explicit, friendlier refusal for the retired brainstorm words —
  `ram`, `cold`, `tiered`, `paged`, `mmap`, `buffer`, and `index` as a
  `resident` value — naming the two real arguments, so vocabulary explored and
  rejected during design does not become folklore in user code.
- [ ] Extend the `@table` line in `dump.ml` to print both properties, so
  `--dump-ast` shows them and the goldens pin them.
- [ ] Add three AST goldens: a bare `@table`, one with `durable: false`, one
  with `resident: keys`. Bless once, by hand.
- [ ] Catalogue the new refusals in `docs/plan/oop-vm/01-error-catalog.md`
  under WO-E102's row — same code, new causes — and update the
  `@table` annotation row in `docs/guides/language-surface.md` §2 to match
  what the parser now accepts.
- [ ] Verify: `just woc-test` — the three new goldens pass and **every
  pre-existing golden is unchanged**. If any existing golden moved, the
  defaults are wrong; fix the defaults, do not rebless.
- [ ] Commit. Draft: `feat(woc): @table durable/resident arguments, defaults preserve behaviour`.

---

## Task 2 — the cross-table check: a durable row may not reference a volatile one

**Files:** modify `compiler/src/types.ml` (the class/field check pass that
already resolves `Ref` field types against the symbol table);
create fixtures under `tests/corpus/compile-fail/`.

**Interfaces:**
- Consumes: Task 1's two `table_cfg` fields, read via `class_info.table`
  (`types.ml:49`).
- Produces: one new `WO-E2xx` code. Pick the next free number by sweeping
  `<stage>_prefix ^ "NN"` across `compiler/src/*.ml` — note a grep for the
  literal `WO-E2NN` finds only comments, which is how ten codes went missing
  before.

- [ ] In the field-checking pass, for every field whose type is a `Ref` to a
  declared table class: if the *declaring* table is durable and the *referenced*
  table is not, report the new code at the field's own position. The message
  must name both classes and say why — a persistent row cannot hold a foreign
  key into a table that evaporates on restart, and FK-restrict cannot save it.
- [ ] Apply the same check to a `backlink` field, whose inverse is a `ref` and
  therefore has the same hazard in the other direction.
- [ ] Leave the reverse direction legal: a volatile table may reference a
  durable one. The dangling case only exists when the survivor points at the
  casualty.
- [ ] Catalogue the code with its message and a worked example.
- [ ] Add two `compile-fail` fixtures: a durable table with a `ref` into a
  volatile one, and the `backlink` twin. Each pins the exact code.
- [ ] Add one `run` fixture proving the legal direction still compiles and
  runs, so the check cannot be over-broad without a gate noticing.
- [ ] Verify: `just woc-test`, `just oop-e2e` — the new fixtures pass, nothing
  else moved.
- [ ] Commit. Draft: `feat(woc): refuse a durable ref into a volatile table`.

---

## Task 3 — format: the class descriptor carries both properties

**Files:** modify `runtime/src/wob.h` (`WOB_VERSION` at :16, `wo_classdesc` at
:547-563), `runtime/src/loader.c` (descriptor parse and validation),
`compiler/src/emit.ml` (descriptor emission), `compiler/src/disasm.ml`,
`docs/plan/oop-vm/00-wob-format.md`.

**Interfaces:**
- Consumes: Task 1's `table_cfg` fields.
- Produces: two descriptor fields the engine reads in Tasks 4 and 6. Name them
  in `wob.h` and use those exact names downstream.

- [ ] Add both properties to `wo_classdesc` at `wob.h:547-563`, in the house
  comment style, positioned so the struct's existing field order and any
  alignment assumption in `wo_obj_size` (:567) are preserved.
- [ ] Bump `WOB_VERSION` at `wob.h:16` to 7, extending the existing comment to
  say what v7 adds — the file's comment history is the format's changelog.
- [ ] Emit both properties from `emit.ml` where the class descriptor is
  written, defaulting exactly as Task 1 does.
- [ ] Extend the loader's descriptor validation to bounds-check both fields and
  **refuse the meaningless combination** — not durable and not fully resident —
  at load time. The loader's contract is that whatever it accepts, the
  interpreter may trust; this combination must never reach the engine.
- [ ] Confirm an image built before this change is refused on version rather
  than misread. This is what the version bump is for.
- [ ] Print both properties in `--dump-bc` (`disasm.ml`) so the corpus can pin
  them from the compiler side.
- [ ] Update `00-wob-format.md`'s descriptor section and its version history in
  this commit.
- [ ] Verify: `just woc-build`, `just wovm-build`, `just wovm-test`,
  `just oop-e2e` all green; a stale `.wob` is rejected clearly.
- [ ] Commit. Draft: `feat(wob): v7 — class descriptor carries durability and residency`.

---

## Task 4 — engine: a volatile table skips the WAL

**Files:** modify `database/src/db.c` (the three WAL-append sites at :26-32,
:48-51, :71-74), `database/src/wal.c` (replay), `database/src/table.h` /
`table.c` if the per-table property needs to reach the choke point;
create fixtures under `tests/corpus/run/`.

**Interfaces:**
- Consumes: Task 3's descriptor fields.
- Produces: the observable behaviour Task 7's gate asserts — no WAL growth for
  a volatile table, and an empty table after restart.

- [ ] Reach the per-table durability property at the three append sites in
  `db.c`. Each already guards on `vm->rt.wal` being non-null; the condition
  becomes "a WAL is open **and** this table is durable". Do not add a second
  choke point — `database/src/CODE-LOGIC.md` names these as the only places
  storage may be staged, and that invariant is worth more than the convenience.
- [ ] Confirm the ack path is unchanged for durable tables: RAM applied, record
  staged, one commit before the ack, and a failed commit still removes the row
  so RAM never claims what disk never acknowledged (the comment at `db.c:29-31`
  states this contract — preserve it exactly).
- [ ] Make replay skip records belonging to a class whose descriptor now says
  volatile. A WAL written when a table was durable and replayed after the
  source changed is a real case; silently resurrecting rows into a table
  declared not to have any is worse than refusing.
- [ ] Decide and implement the mismatch policy for that case: refuse at startup
  naming the class, rather than skipping silently. A silent skip is a data-loss
  surprise; the spec's Migration section commits to refusing on mismatch.
- [ ] Add a `run` fixture: two tables, one durable and one volatile, inserts
  into both, and a second process run proving the durable rows replayed and the
  volatile table is empty.
- [ ] Verify: `just oop-e2e`, `just employee`, `just db-actor` green; measure
  that a volatile table's inserts produce no WAL growth (compare file size,
  do not assert it).
- [ ] Commit. Draft: `feat(db): durable:false skips the WAL append and replay`.

---

## Task 5 — the `resident: keys` read path (split into 5a–5d)

> **SPLIT 2026-08-27, after 5a shipped.** This task was written as if the
> storage side were plumbing on existing functions. It is not, and that was
> measured rather than guessed: `wo_row_ptr` returns a `db_row *` into a slab
> and has **11 call sites**; `table.c` has 37 slab references; the query path
> walks slabs directly (`db.c:105-181`); `enc_val` serialises *from* the slab,
> so an insert must materialise, append, commit and only then drop the payload;
> and **no operation exists that drops a row's payload while keeping its index
> entries** — `wo_row_remove` removes from the indexes too.
>
> Note this is the *opposite* half from the earlier retraction. The record
> FORMAT genuinely needed nothing (retracted, correctly). The record STORAGE
> genuinely is deep, and leaving the step list reading as light plumbing was
> the residual error.
>
> | Sub-task | Scope | State |
> | --- | --- | --- |
> | **5a** | `wo_wal_next_offset` — exact record offsets, unit-proven | ✅ landed `ac7d8af` |
> | **5b** | `wo_wal_read_row_at` — materialise a row from an offset into VM values. **Zero storage change**, so it is additive and independently testable | this section |
> | **5c** | the id→offset map for `resident: keys` classes, plus the missing drop-payload-keep-index operation | not started |
> | **5d** | rewiring `wo_row_ptr`'s call sites, the slab scans, and `@unique`/FK across the residency boundary | not started |
>
> Only 5b is described below. 5c and 5d need their own task write-ups once 5b
> has shown what the read path actually costs.


> **Retraction, 2026-08-26.** This plan originally had a Task 5 that rewrote
> `db_val_encode`/`db_val_decode` into a "self-contained, offset-based" record
> format, described as the iteration's one real rewrite. **That task was
> fictional and has been deleted.** `table.c`'s `db_val_encode` builds the
> *in-memory slot*; the *file* record is a separate encoding in `wal.c`, and it
> has been flat since iteration 9: `enc_val` inlines every kind recursively with
> no pointer anywhere, `dec_val` reads it back into fresh engine values, a record
> is `WO_WAL_INSERT | class_id | id | <value per field>` inside the
> `len|crc|payload|mark` frame, and `scan_record(fd, off, …)` already `pread`s
> and CRC-verifies a record at an arbitrary offset. Nothing about the row
> encoding needs to change.
>
> **The real difficulty is offset capture, and it lives in this task.**
> `wo_wal_append_insert` calls `stage()` into a buffer (opened at 1 MiB in
> `main.c:210`), so a record's final file offset is unknown at append time and
> known only when that buffer flushes. Threading an accurate offset back to the
> caller through a buffered writer — correct across a partial flush, a failed
> commit, and a torn tail — is where to expect the bugs.


**Files:** modify `database/src/table.c` / `table.h` (the per-table id map, the
slab path), `database/src/db.c` (insert/read/update/delete), `database/src/wal.c`
(boot map rebuild); create fixtures under `tests/corpus/run/`.

**Interfaces:**
- Consumes: Task 3's descriptor fields, and `wal.c`'s existing `enc_val`/`dec_val`/`scan_record` — see the note below.
- Produces: a table whose rows are not resident, serving reads by offset.

### 5b — read a row from an offset

**Files:** modify `database/src/wal.c` (beside `scan_record` at :272 and
`dec_val` at :167), `database/src/wal.h`; extend `runtime/test/test_wal.c`.

**Interfaces:**
- Consumes: 5a's `wo_wal_next_offset`, plus the already-public
  `wo_val_decode_vm` and `wo_db_val_free` (`table.h:183-187`).
- Produces: `wo_wal_read_row_at`, which 5c's map consumes as its read path.

- [ ] Add `wo_wal_read_row_at` in `wal.c`, mirroring `wo_row_read`'s contract
  (`table.c:721`) but resolving from a file offset instead of the id hash:
  `scan_record` the record, parse the `kind | class_id | id` header, `dec_val`
  each field into engine-owned slots, convert each to a fresh VM value with
  `wo_val_decode_vm`, then free the engine slots. The out-gate rule is
  unchanged — always a copy, never a pointer into anything.
- [ ] Return distinguishable outcomes: 0 ok, -1 no intact record at that
  offset or a record whose kind carries no payload (a REMOVE tombstone), -2
  OOM with `*msg` set. Silently treating a tombstone as a row would be the
  worst failure available here.
- [ ] Free every engine slot on **every** path including the partial-decode
  error path, matching what `dec_val`'s own callers already do at `wal.c:204`.
  ASan is the check, not inspection.
- [ ] Do not change any storage behaviour. Nothing calls this yet; it is
  additive, which is the whole point of separating it from 5c.
- [ ] Unit-test it against 5a's offsets: insert rows of mixed kinds
  (scalar, Text, and a nil Text), record each offset, then read every row
  back **by offset** and deep-compare field by field to what was inserted.
  Include a read at a deliberately wrong offset and at a tombstone, asserting
  the documented refusals rather than a crash.
- [ ] Verify: `make -C runtime test` and `test-iso`, both ASan+UBSan clean;
  `just oop-e2e`, `just residency`, `just employee`, `just db-actor`
  unchanged, since nothing calls the new function yet.
- [ ] Commit. Draft: `feat(db): wo_wal_read_row_at — materialise a row from a log offset`.

## Task 6 — the two runtime refusals

**Files:** modify `runtime/src/main.c` (the `WO_DATA` block at :199-215),
plus wherever per-table accounting lands from Task 6.

**Interfaces:**
- Consumes: Task 3's descriptor fields, Task 5's accounting.

- [ ] **Refuse `durable: true` with no `WO_DATA`.** Today `main.c:199` opens a
  WAL only when the variable is set, and `db.c` skips appends when it is not —
  so a program that declares durability and is given nowhere to put it silently
  loses everything. Refuse at startup, naming the first durable class found.
  This is the single most valuable line in the plan and is independent of
  residency.
- [ ] Provide the escape hatch the refusal implies: a program that genuinely
  wants an ephemeral run must be able to say so, either by declaring its tables
  volatile or by an explicit opt-out flag. Decide which and document it — a
  refusal with no stated way forward is a worse bug than the silent loss.
- [ ] Implement the byte budget: estimated resident footprint across all
  tables, breached loudly with a message naming the largest offending table and
  the exact annotation to add.
- [ ] Default the budget to a fraction of host-detected available memory, **not
  to "none"** — a budget nobody sets cannot produce the diagnostic that is this
  design's main deliverable, and the 120 GB developer would still meet the OOM
  killer. Use a conservative placeholder fraction and mark the value explicitly
  unset-pending in both the code comment and the iteration: the real number
  comes from databasev2 1's swap-onset measurement.
- [ ] Make the accounting's error bound explicit where it is documented. It
  estimates RSS; it is not RSS, and pretending otherwise would make the budget
  untrustworthy the first time someone checked it.
- [ ] Add CLI-smoke coverage for both refusals, including the exit code and the
  first line of stderr — the shape scripts depend on.
- [ ] Verify: `bash runtime/test/cli_smoke.sh`, `just wovm-test`,
  `just employee`, `just db-actor` green; every sample that sets `WO_DATA`
  still runs, and one that does not is now refused or explicitly opted out.
- [ ] Commit. Draft: `feat(rt): refuse silent volatility, and bound resident footprint`.

---

## Task 7 — measure, gate, document, close out

**Files:** modify `scripts/db-bench.py` and `docs/examples/db-bench/`,
`bench/baseline.json`, `docs/plan/perf-targets.md`,
`docs/plan/oop-vm/04-db-binding.md`, `database/src/CODE-LOGIC.md`,
`docs/stories/databasev2/02-table-storage-modes.md`,
`docs/stories/databasev2/00-story.md`, `docs/stories/00-status.md`,
`docs/guides/language-surface.md`.

- [ ] Add db-bench legs for the `resident: keys` read and write paths at a
  size that exceeds resident memory, and a volatile-write leg showing the
  per-table saving from Task 4.
- [ ] Add the resulting rows to `bench/baseline.json` with per-class tolerances
  following iteration 22's policy, and confirm the gate bites on a doctored
  metric — a gate that only prints is not a gate.
- [ ] Publish the read amplification in `perf-targets.md`: how much slower a
  non-resident read is than a resident one, as a number a developer can plan
  around, alongside the measured per-row index footprint.
- [ ] Run the crash battery on a `resident: keys` table: `kill -9` mid-append
  and mid-read, replay, and assert no acked write lost and no row visible
  twice.
- [ ] Update `04-db-binding.md` with the residency contract and the record
  layout, and `database/src/CODE-LOGIC.md` with the encode/decode rewrite and
  the boot-rebuild cost.
- [ ] Rewrite `docs/stories/databasev2/02-table-storage-modes.md` around what
  shipped and remove its supersede banner; correct the track index
  (`00-story.md`), whose §"The lever" still describes the three-mode `cold`
  design the brainstorm replaced.
- [ ] Set the iteration's frontmatter `status: done` and add the board standup
  entry answering the six questions, including which `.dev/reference` projects
  were used.
- [ ] Verify: the whole battery — `just woc-test`, `just wovm-test`,
  `just oop-e2e`, `just oop-accept`, `just employee`, `just db-actor`,
  `just db-bench`, `just web-app`, `just site`, `just linkcheck`.
- [ ] Commit. Draft: `feat(db): table residency — measured, gated, documented`.

---

## Self-review against the spec

Checked section by section; recorded here so a reviewer can see what was and
was not covered.

| Spec section | Task |
| --- | --- |
| Grammar (two arguments, defaults) | 1 |
| The four combinations, incl. the refused one | 1 (parse), 3 (loader) |
| Compiler refusals: bad value, given twice, retired words | 1 |
| Compiler refusal: durable ref into volatile | 2 |
| Format / descriptor / version bump | 3 |
| `durable: false` skips the WAL | 4 |
| Replay skips or refuses on mismatch | 4 |
| Self-contained offset-based records | **none — already exists in `wal.c` since iteration 9. Claim retracted 2026-08-26; see the spec section of the same name.** |
| Read / update / delete / scan / boot for non-resident, incl. offset capture | 5 |
| `@unique` and FK-restrict across the boundary | 5 |
| Startup refusal: durable with no `WO_DATA` | 6 |
| Byte budget, default fraction, breach diagnostic | 6 |
| Proof plan: baseline, amplification, crash battery | 7 |
| Docs: catalog, language surface, binding contract, CODE-LOGIC | 1, 3, 7 |

**Gaps found and closed during review:** the spec's escape hatch for an
intentionally ephemeral run was implied but never stated — added as an explicit
step in Task 6, because a refusal with no way forward is worse than the silent
loss it replaces. The spec's note that databasev2 3's snapshot should persist
the offset map is now a recorded step in Task 5 rather than prose only.

**Deliberately not in this plan:** checkpoint and compaction (databasev2 3),
eviction and a resident row cache (databasev2 5), io_uring on the read path
(databasev2 4's note), per-shard residency for volatile tables (recorded as a
candidate in the spec), and any conversion of an existing dataset between
settings — the spec commits to refusing on mismatch, not converting.