docs: amend principle 7 — the log is authoritative, residency is declared

- driving case: a 120 GB order table on a 32 GB host. Not a tuning problem;
  no eviction policy fixes it. Developer accepted reconsidering the principle
- principle 7 rewritten: durability half UNCHANGED and unconditional
  (WAL-logged, fsync before ack, CRC-dropped torn tail); residency half
  demoted from law to per-table declaration. Old wording quoted in place so
  the amendment is legible, with the reason: a doctrine a real workload
  cannot satisfy gets ignored, and the failure it produced was an OOM kill
- spec: docs/superpowers/specs/2026-08-26-table-residency-design.md
  One log-structured engine — the WAL already holds every row, so keep an
  in-RAM id->offset map and pread rows back. No second engine, no user-space
  row cache (the kernel page cache is the hot copy, which is already this
  repo's stated position and why it avoids O_DIRECT)
- arithmetic that makes it work: 240M rows x 16 B of index = ~3.8 GB
  resident in 32 GB. Indexes stay resident, rows do not. Buys ~2 orders of
  magnitude, not infinity — stated plainly in the spec
- grammar: two optional keys, `durable: true|false` and `resident: all|index`,
  both defaulting to today's behaviour, so all 28 existing @table
  declarations compile untouched and no golden is reblessed
- rejected, with reasons recorded: mmap (rows are pointer-bearing —
  table.c returns (uintptr_t)t as the slot word), buffer pool (the Rust-era
  phase-12 design that died with that track), paged B-tree (stays rejected),
  a three-valued enum, automatic spill, disk-backed-by-default
- self-review caught the budget defaulting to "none" while promising the ERP
  developer a diagnostic instead of the OOM killer — contradiction fixed:
  the budget defaults to a fraction of host memory, and its value comes from
  databasev2 1's swap-onset measurement
- live docs that contradicted the amendment updated (subagent doctrine,
  its guide, discarded.md's two rows, iteration 04's read claim, 07, 38);
  dated specs/plans left as records. linkcheck 0 broken / 0 anchors

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
shoney.arickathil 2026-08-26 22:42:27 +02:00
parent 746dc2b42b
commit da30aa6527
10 changed files with 346 additions and 20 deletions

View file

@ -14,7 +14,8 @@ You are the database engineer for writeonce's embedded engine.
Doctrine (non-negotiable):
- C11 + libc only. No new dependencies, no atomics on the data path.
- RAM is authoritative; the WAL makes it durable. An ack means the
- The log is authoritative; residency is a declared per-table policy
(principle 7, amended 2026-08-26). An ack means the
commit fsynced. Replay is whole-or-not-at-all; torn tails drop.
- The engine and the VM heap are two memory worlds crossed only by
copy (the out-gate: wo_val_decode_vm always copies; rows never hold

View file

@ -68,17 +68,42 @@ binary embeds its own source, so prod is always self-describing.
events; a database that is also the app must not blink.
*Enforced by:* [the blue-green spec](superpowers/specs/2026-08-03-blue-green-vm-design.md).
## 7. RAM is authoritative; the WAL makes it durable
## 7. The log is authoritative; residency is a declared per-table policy
**Amended 2026-08-26.** This principle read "RAM is authoritative; the WAL
makes it durable. All reads serve from memory." The durability half was never
under strain and is unchanged. The residency half was false for a real
workload, so it is now a declaration rather than a law.
**Durability, unconditional:** every mutation is WAL-logged and fsynced before
acknowledgment; boot replays the log; a torn tail is dropped whole by CRC; an
ack means the commit reached disk. Mirrors (Postgres) are reconstructible
backups that reads and acks never depend on. None of this is per-table and
none of it is negotiable.
**Residency, declared:** what a table keeps in memory is stated at the
declaration site. The default keeps every row resident and serves reads at
memory speed. A table that cannot fit says so, and then only its indexes are
resident while rows are read from the log by offset — the kernel page cache is
the hot copy, which is why the engine uses `pread` and deliberately not
`O_DIRECT`.
*Why the amendment:* the original wording is right for a knowledge-management
app and simply false for a 120 GB order table on a 32 GB host. A doctrine a
real workload cannot satisfy does not get followed, it gets ignored — and the
failure it produced was an OOM kill, which is the least debuggable outcome
available. The fix keeps one storage engine and one source of truth: the log
*is* the database, and RAM is how much of it you choose to serve fast. What was
rejected in 2026-08-18 and stays rejected is a *second* engine — a paged
B-tree with its own buffer pool ([`plan/discarded.md`](plan/discarded.md)).
Reading rows from the log we already write is not that.
All reads serve from memory. Every mutation is WAL-logged and fsynced
before acknowledgment; boot replays the log. Mirrors (Postgres) are
reconstructible backups that reads and acks never depend on.
*Why:* one source of truth with predictable latency; durability is a
sequential append, not a storage engine bolted to the side.
*Enforced by:* [the db-engine binding plan](superpowers/plans/2026-08-01-db-engine-binding.md)
(typed WAL + boot replay, shipped); the mirror-is-backup doctrine is
recorded in [`plan/discarded.md`](plan/discarded.md) (the Rust-era WAL
and mirror plans 11/16 were removed with that track 2026-08-18).
(typed WAL + boot replay, shipped); the residency declaration and its
enforcement are [databasev2 2](stories/databasev2/02-table-storage-modes.md);
the mirror-is-backup doctrine is recorded in
[`plan/discarded.md`](plan/discarded.md) (the Rust-era WAL and mirror plans
11/16 were removed with that track 2026-08-18).
## 8. Samples force the grammar

View file

@ -42,7 +42,8 @@ You are the database engineer for writeonce's embedded engine.
Doctrine (non-negotiable):
- C11 + libc only. No new dependencies, no atomics on the data path.
- RAM is authoritative; the WAL makes it durable. An ack means the
- The log is authoritative; residency is a declared per-table policy
(principle 7, amended 2026-08-26). An ack means the
commit fsynced. Replay is whole-or-not-at-all; torn tails drop.
- The engine and the VM heap are two memory worlds crossed only by
copy (the out-gate: wo_val_decode_vm always copies; rows never hold

View file

@ -43,7 +43,7 @@ Status board: [`00-status.md`](../stories/00-status.md) · Doctrine: [`../00-pri
| **Self-hosted `.wo` deploy logic** | Bootstrap problem — the deploy path cannot be written in the language whose deployment it implements. Recorded as a much-later possibility. |
| **Script-based / destructive schema migrations (v1)** | Additive-only auto-diff is what makes rollback unconditional: the previous version ignores fields and classes it never knew. Destructive changes reject at propose time. |
| **Portability abstractions over Linux** | Targeting one kernel lets the runtime use its sharpest primitives directly instead of the lowest common denominator. Principle 9. |
| **Mirrors on the commit path** | The Postgres mirror is a reconstructible backup. RAM is authoritative; reads and acks never depend on it. Principle 7. |
| **Mirrors on the commit path** | The Postgres mirror is a reconstructible backup. Reads and acks never depend on it — the log is authoritative. Principle 7. |
## Process / docs
@ -72,7 +72,7 @@ goes to find what took each one's place.
| --- | --- |
| `09-concurrency-scaleout.md` | [`08-shard-actor-runtime.md`](../stories/language-runtime-database/08-shard-actor-runtime.md) + [`11-fibers.md`](../stories/language-runtime-database/11-fibers.md) — the arc, landed 2026-08-21 |
| `10-storage-foundations.md`, `11-wal-and-recovery.md` | [`09-database-engine.md`](../stories/language-runtime-database/09-database-engine.md) (typed WAL + replay) and [`22-durability-throughput-scale.md`](../stories/language-runtime-database/22-durability-throughput-scale.md) (the measurements) |
| `12-engine-disk-cutover.md` | Nothing — RAM stays authoritative by doctrine (principle 7). The disk story is the WAL; reclamation is [`databasev2 3, WAL checkpoint`](../stories/databasev2/03-wal-checkpoint.md) |
| `12-engine-disk-cutover.md` | **Partly revisited 2026-08-26.** The disk story is still the WAL and a paged B-tree with its own buffer pool stays rejected. But principle 7's *residency* half was amended: a table may now declare that only its indexes are resident and its rows are read from the log by offset — [databasev2 2](../stories/databasev2/02-table-storage-modes.md). Reclamation remains [`databasev2 3, WAL checkpoint`](../stories/databasev2/03-wal-checkpoint.md) |
| `13-class-model-live-pricing.md` | [`09b-table-relations-query.md`](../stories/language-runtime-database/09b-table-relations-query.md) — `@table`, `ref`/`backlink`, the compiler-checked query surface |
| `07-inotify-content-watcher.md` | [`07-logwatcher-proof.md`](../stories/language-runtime-database/07-logwatcher-proof.md) — the log-watcher sample polls via `fs.stat`; inotify was never surfaced as a builtin |
| `08-sendfile-static-assets.md` | Nothing. `sendfile` is not exposed; static assets are served as `Text` through `net.write` |

View file

@ -604,7 +604,7 @@ story file); slots in when scheduled — nothing in the chain depends on it.
### ▸ databasev2 — the database beyond RAM
New 2026-08-26. **The problem:** RAM is authoritative (principle 7) and nothing
New 2026-08-26. **The problem:** rows were resident unconditionally and nothing
declares a budget. Rows live in `malloc`'d slabs whose addresses are stable
forever; there is no eviction, spill or paging anywhere in `database/src/`; the
WAL never checkpoints so boot replays all history; and durability is one

View file

@ -19,6 +19,14 @@ status: refine
> global switch forces "everything is precious" or "nothing is", and the
> developer pays for the wrong one either way.
> **⚠ Superseded in part, 2026-08-26.** The brainstorm settled on a different
> shape than this document describes: one log-structured engine where the WAL
> *is* the row store, with residency declared per table (`resident: all` /
> `resident: index`) rather than a three-valued `mode:` enum including `cold`.
> Principle 7 was amended accordingly — the log is authoritative, residency is
> the declaration. This file is rewritten once the grammar is approved; read the
> [track index](00-story.md) and `docs/00-principles.md` §7 as current.
## Goals
- **Move the storage decision to the declaration site.** `@table(mode: ...)`,

View file

@ -93,9 +93,14 @@ chain: 5
- **io_uring for the network/accept path.** This iteration is the WAL write
path only; the socket side is the shard-actor runtime's and the network
layer's concern.
- **io_uring for reads.** RAM is authoritative — reads never touch a
descriptor (phase-B doctrine), so there is nothing to accelerate on the
read path. This is a write-durability optimization, full stop.
- **io_uring for reads.** For a fully-resident table reads never touch a
descriptor, so there is nothing to accelerate on the read path. This is a
write-durability optimization, full stop. **Note (2026-08-26):** principle 7's
residency half was amended, so a table declaring `resident: index`
([databasev2 2](02-table-storage-modes.md)) *does* `pread` rows from the log —
and accelerating that read path with io_uring becomes a real, separate
question. It is not this iteration's, and it should not be folded in: this
one is about the commit path and its acceptance is a durability number.
- **Registered buffers / fixed files / SQPOLL tuning** beyond what the
benchmark shows is worth it. Start with the plain submit/complete model;
add ring features only when 22's number says a specific one pays.

View file

@ -17,7 +17,7 @@ status: refine
> **Inserted 2026-08-22** (developer ask: "can the persistent db be in
> file.db form?"). The truth is already almost there: `WO_DATA=<dir>`
> holds exactly ONE file (`shard-0.wal`) — the entire persistent state,
> since RAM is authoritative and no data pages exist. This iteration
> since the log is the whole store and no data pages exist. This iteration
> makes the surface say so: point `WO_DATA` at a file and THAT file is
> the store. Small, driver-only, independent of the concurrency chain.
@ -50,7 +50,7 @@ status: refine
## Out Of Scope
- A paged database file (SQLite's shape) — RAM is authoritative; the
- A paged database file (SQLite's shape) — still rejected; the
disk story is the WAL, full stop.
- Checkpoint/compaction — [iteration 32](03-wal-checkpoint.md)'s; its
rename-swap (write snapshot+tail to a NEW file, fsync, `rename()`

View file

@ -116,7 +116,7 @@ status: refine
- **A plugin/app ecosystem.** In-runtime recompile is
[iteration 26](26-blue-green-deploy.md)'s; nothing here loads
code at run time.
- **Scale claims.** RAM is authoritative (64 MiB arena by default,
- **Scale claims.** Rows are resident by default (64 MiB arena default,
`WO_HEAP_MB`), so this is a team-scale platform and the README states the
row ceiling it was measured at.

View file

@ -0,0 +1,286 @@
# databasev2 2 — table residency: the log as row store (for review)
> **Status:** design approved in brainstorm 2026-08-26, pending review of this
> document. Implements [databasev2 2](../../stories/databasev2/02-table-storage-modes.md),
> which is rewritten to match once this is approved. Amends
> [principle 7](../../00-principles.md) — already applied.
>
> Driving requirement, from the developer: **a 120 GB order table on a 32 GB
> host.** Not a tuning problem; no eviction policy fixes it. Small data-driven
> applications are well served by today's resident default and must not regress.
## Decisions taken (the brainstorm's forks, settled)
| Fork | Decision |
| --- | --- |
| One enum or two keys | **Two keys.** `durable:` and `resident:` answer two different developer questions ("do I need this after a restart?", "does it fit in RAM?"). One enum forces a name for each *combination*, which is what made a third value unreadable. |
| Mode vocabulary | **`resident: all \| index`** and **`durable: true \| false`**. No `cold`, `tiered`, `paged`, `mmap` or `buffer` in the grammar. |
| Optional or mandatory | **Optional, both default to today's behaviour** (`durable: true`, `resident: all`). All 28 existing declarations compile unchanged; no goldens reblessed. |
| Which storage architecture | **One engine, log-structured.** The WAL already holds every row; keep an in-RAM id→offset map and read rows back with `pread`. No second engine. |
| Row cache | **None in user space.** The kernel page cache is the hot copy — the repo's own stated position in `exploration/postgresql/buffer-and-checkpoint.md`: "`pread` against an fd that already has its page cached is a memcpy… the page cache is the one cache we want", and the reason the engine avoids `O_DIRECT`. |
| `@unique` on a non-resident table | **Allowed; its index is unconditionally resident.** Settled here rather than deferred — see Constraints. |
| Budget unit | **Bytes** (estimated resident footprint). Rows is the meaningless unit: a text-heavy row and an Int-only row differ by an order of magnitude, so a row count cannot bound RAM. |
| Rejected architectures | `mmap` and a buffer pool stay out — see Alternatives rejected. `discarded.md`'s paged-engine rejection is amended to *partly revisited*, not reversed. |
## The problem, read off the engine
Facts, each verified in source rather than assumed:
- Rows live in `malloc`'d slabs of `DB_SLAB_ROWS` (256), allocated as a table
grows and freed only at table teardown. The free-slot list recycles removed
slots, so a delete-heavy table plateaus; a growing table only grows.
- **The ceiling is process RSS and nothing declares it.** `WO_HEAP_MB`
(default 64 MiB) bounds the VM object arena; table storage is separate
`malloc`. No knob says "this database may use at most N".
- No eviction, spill, paging or LRU exists anywhere in `database/src`.
- **Durability is process-global.** `main.c` opens one `shard-0.wal` when
`WO_DATA` is set. `db.c` guards every WAL append with a null check on
`vm->rt.wal`, so with no `WO_DATA` **every table is silently volatile** — a
program can declare nothing and lose everything.
- An allocation failure is clean: every `malloc` in the row encoder is checked
and `DB_ERR_OOM` maps to `WO_T_OOM`, a catchable trap. The dangerous exit is
the one *before* that — swap thrash, which carries no error signal at all.
The measurements that bound the design, from iteration 22: durable inserts
≈4.5k/s against RAM ≈297k/s (the 66× fsync gap); reads 1.3M ops/s at p50 1µs
after the index probe landed. The read number is what a resident table buys and
what must not regress for tables that keep it.
### Why the arithmetic works
A 120 GB table at ~500 B/row is ≈240M rows. An id→offset entry is 16 bytes, so
the resident index is **≈3.8 GB** — comfortable inside 32 GB with room for two
secondary indexes of similar cost. That ratio is the whole design: **indexes
stay resident, rows do not.** It buys roughly two orders of magnitude of table
size, not infinity, and the spec says so plainly because a design sold as
unlimited gets deployed as if it were.
## The design
### Grammar
Two new `@table` arguments, both optional. The parser's argument loop already
matches `name` and `index` and rejects anything else with a catalogued
diagnostic; these are two more arms and an updated message. `Ast.table_cfg`
grows two fields beside `table_name` and `indexes`.
| Argument | Values | Default | Meaning |
| --- | --- | --- | --- |
| `durable` | `true`, `false` | `true` | `false` skips the WAL append entirely: no record, no fsync, ack from RAM, table empty after restart. |
| `resident` | `all`, `index` | `all` | `index` keeps the id map and every secondary index in RAM; rows are read from the log by offset. |
`true`/`false` are already keyword tokens; `all`/`index` are parsed as the same
bare identifiers the `index:` argument's column list already accepts. No lexer
change.
**One wart, surfaced rather than buried:** `resident: index` puts the word
`index` in value position while `index:` is also a key, so
`@table(index: [customer], resident: index)` reads awkwardly on first
encounter. The parser distinguishes them structurally and there is no
ambiguity, but a reviewer may prefer `resident: keys` or `resident: index_only`.
Flagged for the review of this document; the mechanism is unaffected either
way.
### The four combinations
| `durable` | `resident` | Status |
| --- | --- | --- |
| `true` | `all` | Today's behaviour. The default. Reads at memory speed. |
| `false` | `all` | Volatile scratch: sessions, rate-limit counters, idempotency keys. Skips the 66× fsync cost. What porch 1–3 need. |
| `true` | `index` | The 120 GB case. Rows in the log, indexes resident, `pread` on read. |
| `false` | `index` | **Refused at compile time.** Rows would have nowhere to be read from. |
### Read, write and recovery paths
- **Insert** — unchanged for `resident: all`. For `resident: index`: encode and
append the record as today, then record id→offset in the resident map instead
of retaining the row in a slab. The WAL append is already the durable write;
this stops discarding its payload.
- **Read by id** — resident map lookup, then `pread` at the offset, verify CRC,
decode into fresh VM values. The decode path already exists
(`wo_val_decode_vm` always copies; rows never hand out interior pointers), so
the change is where the bytes come from.
- **Update** — append a new record, repoint the offset. The superseded record
becomes garbage, reclaimed by the checkpoint.
- **Delete** — append a tombstone, drop the id from the map and every index.
- **Scan** — a sequential walk of the log, which is the case log-structured
storage is best at. Cost changes from memory-speed to sequential-disk; the
query surface is unchanged in *meaning* and materially different in *cost*.
- **Boot** — replay rebuilds the offset map by scanning the log. Correct, and
O(entire history), which is the honest cost of shipping this before
[databasev2 3](../../stories/databasev2/03-wal-checkpoint.md).
### Row encoding: the one real rewrite
A row slot today holds raw pointers. `table.c`'s `WO_K_TEXT` case allocates a
`db_text` and returns its address as the slot word; owned, multi and map do the
same. Pointers minted by a dead process are meaningless in a file, so the
on-disk record must be **self-contained and offset-based**: every heap value
inlined into the record with internal references expressed as offsets from the
record's own start.
This is confined to `db_val_encode`/`db_val_decode` and the record framing. It
is the substantive engineering in this iteration and the place to expect the
bugs. `wal.c` already frames records as `len|crc|payload|mark`, so the framing
exists; what changes is that the payload must be readable standalone rather than
only replayable.
### Constraints across the residency boundary
- **`@unique` is allowed, and its index is unconditionally resident.** A shadow
check cannot scan slabs that are not there, so correctness requires the unique
column's index in RAM. Cost is one hash entry per row — the same order as the
id map, so a unique column roughly doubles the resident index. Stated at the
declaration so the developer can see what they bought. Silently checking only
resident rows is the one outcome that must never ship.
- **Foreign-key restrict works unchanged.** It is a secondary-index probe, and
secondary indexes are resident.
- **`ref` navigation works unchanged**, at the cost of a `pread` per hop.
- **A `resident: all` table may hold a `ref` into a `resident: index` table**
and vice versa — both are durable, so neither evaporates. This is the case
that a `durable: false` table genuinely breaks, below.
### What the compiler enforces
Four refusals, each a catalogued `WO-E1xx` diagnostic added in the same change
as the code — not afterwards:
1. `durable: false` with `resident: index` — the meaningless combination.
2. An unknown value for either argument, or either argument given twice.
3. **A `durable: true` table holding a `ref` into a `durable: false` table.**
A persistent row cannot reference one that evaporates on restart; FK restrict
cannot save it and the dangling reference is provable from the class table.
This is the highest-value check in the iteration.
4. A `cold`, `tiered`, `ram` or other retired mode word — refused with a message
naming the two real arguments, so the vocabulary explored during the
brainstorm does not become folklore.
### What the runtime enforces
- **`durable: true` with no `WO_DATA` is a startup refusal.** Today this
combination silently loses everything, which is the worst failure mode in the
current engine. A program that declares durability and is given nowhere to put
it must not start. This is independent of residency and is arguably the most
valuable single line in the spec.
- **A byte budget that exists by default, breached loudly.** The budget bounds
estimated resident footprint across all tables. On breach the program refuses
with a message naming the largest offending table and the exact annotation to
add, so the 120 GB developer meets a diagnostic at 32 GB rather than the OOM
killer.
**The default must not be "no budget"** — that was a contradiction in the
first draft of this spec: a budget nobody sets cannot produce the diagnostic
that is this design's main deliverable, and the ERP developer would still meet
the OOM killer. So the default is a **fraction of host-detected available
memory**, overridable by an environment variable and by a per-program
declaration. Choosing that fraction is the one number this spec cannot supply:
it comes from [databasev2 1](../../stories/databasev2/01-ram-ceiling-measurement.md)'s
swap-onset measurement, which is why 1 sequences before 2. Until 1 lands,
implement the mechanism with a conservative placeholder fraction and treat the
value as unset rather than settled.
Consequence to accept deliberately: a program that today grows past that
fraction and survives on a large host will now refuse. That is the intended
behaviour change — it converts an invisible slide into swap into a startup
error — but it *is* a behaviour change, and it is the second of the two
breaks listed under Migration.
- `WO_DATA` remains the data-directory location. It stops being the durability
switch; the declaration is.
### Format
The class descriptor carries both properties, so the runtime never re-derives
them. This moves `WOB_VERSION` (currently 6) and the format contract in the same
commit as the code. An older image is refused on version rather than misread.
## Why this shape
- **It keeps one engine and one source of truth.** The log *is* the database.
Nothing here adds a second storage system, which is what the 2026-08-18
rejection was actually about.
- **It costs nothing for tables that do not use it.** A `resident: all` table
takes the same path it takes today. The 1.3M ops/s read baseline is the
regression gate, and a measurable regression there is grounds to reject the
implementation rather than tune it.
- **The failure mode becomes a diagnostic.** Two of the three exits
characterised in databasev2 1 — swap thrash and the OOM killer — are replaced
by a refusal that names the fix.
- **It is declared, not automatic.** No threshold heuristic, no performance
cliff the compiler cannot explain. Consistent with a language whose thesis is
that the compiler tells you the truth.
## Alternatives rejected
| Alternative | Why not |
| --- | --- |
| **`mmap` the row file** | Requires offset-based rows *and* gives up precise ack-after-fsync for the kernel's flush schedule. The repo's own mmap study only ever proposed it read-only for segment lookups. If rows become self-contained anyway, mmap is a possible later optimisation of the read path — recorded, not adopted. |
| **Buffer pool with dirty-page tracking** | This is the Rust-era phase-12 design in `exploration/postgresql/buffer-and-checkpoint.md` (`CachedRow { bytes, dirty }`, `WO_CACHE_ROWS` LRU) that died with that track. It duplicates the kernel page cache, and the page cache is explicitly the cache this project wants. |
| **Paged B-tree engine** | Rejected 2026-08-18 and still rejected. Reading rows from the log we already write is not this. |
| **A three-valued `mode:` enum** | Needs a name per combination. The brainstorm demonstrated that the third name is unwriteable before its mechanism is decided. |
| **Automatic spill at a threshold** | No annotation needed, but unpredictable cliffs and magic the compiler cannot explain. |
| **Disk-backed by default** | Costs every application the 1µs read, including the small ones explicitly said to be well served today. |
## Proof plan
Acceptance is the story's Given/When/Then list; this is how each is exercised.
- **Compatibility** — all 28 existing `@table` declarations compile untouched;
`just employee`, `just db-actor`, `just web-app`, `just site`, `oop-accept`
unchanged; no golden reblessed.
- **Volatility** — a `durable: false` table produces no WAL growth (measured,
not asserted) and is empty after restart while durable siblings replay intact.
- **Residency correctness** — a `resident: index` table larger than the
configured budget returns every row correctly by id, byte-identical including
every heap-valued column, and scans in full.
- **Constraints across the boundary** — `@unique` refuses a duplicate whose
conflicting row is not resident; FK restrict refuses a delete whose only
referrer is not resident. Corpus fixtures, both.
- **Compiler refusals** — a `compile-fail` fixture per diagnostic, each pinning
the exact code.
- **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup;
a budget breach names the table and the annotation.
- **Crash safety** — `kill -9` mid-append and mid-checkpoint on a
`resident: index` table; replay loses no acked write and no row appears twice.
- **Performance** — new baseline rows for the `resident: index` read path with
its amplification versus resident, published in `perf-targets.md` as a number
a developer can plan around; and a regression check that resident tables did
not move.
- **Sanitisers** — ASan on the new decode path, which is where the bugs are.
## Out of scope
- **Checkpoint and compaction** — [databasev2 3](../../stories/databasev2/03-wal-checkpoint.md).
This spec's boot cost is O(history) until 3 lands, and 3's snapshot should
persist the offset map so boot stops rescanning. That coupling is stated in
both documents.
- **Eviction policy and a resident row cache** — [databasev2 5](../../stories/databasev2/05-bounded-tables-eviction.md).
This design needs neither: rows are either all resident or none are.
- **io_uring on the read path** — a real question that only exists after this
lands; noted in [databasev2 4](../../stories/databasev2/04-io-uring-commit.md),
not folded into it.
- **Per-shard residency** — the owner shard owns the store and the log. A
`durable: false` table arguably need not live on the owner at all, which would
be dramatically faster and a different consistency story. Recorded as a
candidate, deliberately not decided here.
- **Migrating an existing dataset between settings**, and a WAL written when a
table had different settings: refuse clearly on mismatch, do not convert.
- **`transaction { }` and `@table` feature flags** — language iteration 18,
approved spec, left whole.
## Migration
Nothing to convert. Both arguments default to present behaviour, so every
existing program, manifest, corpus fixture and golden compiles and runs
unchanged.
**Two deliberate behaviour changes**, both converting a silent failure into a
loud one, and both listed here so neither arrives as a surprise:
1. `durable: true` (the default) with no `WO_DATA` becomes a startup refusal.
Today it silently discards every write.
2. Total estimated resident footprint crossing the default budget fraction
becomes a startup or insert refusal. Today the program slides into swap with
no signal and is eventually killed.
Both are opt-out-able by explicit declaration. Neither is a data-format change,
so a rollback is a binary swap with no migration.