docs: name the residency value keys, not index (review change)

- `resident: all | keys` replaces `resident: all | index`. Two reasons beyond
  taste: it kills the collision with the `index:` argument
  (`@table(index: [customer], resident: index)` read badly), and it puts both
  values on ONE axis — each now answers "what row data stays resident",
  where `all`/`index` mixed a quantity with a structure name
- accurate as well as clearer: what stays resident is the id->offset map, the
  secondary indexes and the unique shadows — all key structures; row payloads
  are exactly what leaves. `resident: none` was rejected as overclaiming,
  since the indexes very much are resident
- checked for collisions: neither `all` nor `keys` is a keyword or a builtin
  (`key_at`/`val_at` exist, bare `keys` does not)
- the spec's wart note became a recorded decision; the rejected spelling is
  kept quoted so the rationale still reads
- fixes a bug I introduced in the 2026-08-26 track move: all six moved
  iterations carried a banner reading "Part of [Story — the database beyond
  RAM]" whose link pointed at the LANGUAGE arc — correct target, lying text,
  the exact failure mode the link audit warned about. Banners now point at
  the databasev2 story, and the original "Part of" line says plainly which
  track the iteration was authored in before the move
- linkcheck 0 broken / 0 anchors

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
shoney.arickathil 2026-08-26 22:47:24 +02:00
parent da30aa6527
commit 566559cf70
8 changed files with 39 additions and 31 deletions

View file

@ -22,7 +22,7 @@ status: refine
> **⚠ Superseded in part, 2026-08-26.** The brainstorm settled on a different > **⚠ Superseded in part, 2026-08-26.** The brainstorm settled on a different
> shape than this document describes: one log-structured engine where the WAL > shape than this document describes: one log-structured engine where the WAL
> *is* the row store, with residency declared per table (`resident: all` / > *is* the row store, with residency declared per table (`resident: all` /
> `resident: index`) rather than a three-valued `mode:` enum including `cold`. > `resident: keys`) rather than a three-valued `mode:` enum including `cold`.
> Principle 7 was amended accordingly — the log is authoritative, residency is > Principle 7 was amended accordingly — the log is authoritative, residency is
> the declaration. This file is rewritten once the grammar is approved; read the > the declaration. This file is rewritten once the grammar is approved; read the
> [track index](00-story.md) and `docs/00-principles.md` §7 as current. > [track index](00-story.md) and `docs/00-principles.md` §7 as current.

View file

@ -9,11 +9,12 @@ chain: 6
# databasev2 3 — WAL checkpoint: disk space reclamation and bounded replay # databasev2 3 — WAL checkpoint: disk space reclamation and bounded replay
> **Moved 2026-08-26** from the language track, where this was iteration 32. > **Moved 2026-08-26** from the language track, where this was iteration 32.
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by > Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by
> the move; its dependencies are restated in that track index. > the move; its dependencies are restated in that track index.
> Format: `product/story-iteration-template`. Part of > Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). > [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md)
> — the track this iteration was authored in before the 2026-08-26 move.
> >
> **Inserted 2026-08-21** (stage-3 guarantee refinement found the hole): > **Inserted 2026-08-21** (stage-3 guarantee refinement found the hole):
> the WAL is append-only FOREVER — no checkpoint, no truncation exists > the WAL is append-only FOREVER — no checkpoint, no truncation exists

View file

@ -9,11 +9,12 @@ chain: 5
# databasev2 4 — io_uring group-commit write path # databasev2 4 — io_uring group-commit write path
> **Moved 2026-08-26** from the language track, where this was iteration 23. > **Moved 2026-08-26** from the language track, where this was iteration 23.
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by > Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by
> the move; its dependencies are restated in that track index. > the move; its dependencies are restated in that track index.
> Format: `product/story-iteration-template`. Part of > Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). > [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md)
> — the track this iteration was authored in before the 2026-08-26 move.
> >
> **Inserted 2026-08-15.** The write-path optimization, and deliberately the > **Inserted 2026-08-15.** The write-path optimization, and deliberately the
> LAST database performance iteration: it only earns its complexity once > LAST database performance iteration: it only earns its complexity once
@ -96,7 +97,7 @@ chain: 5
- **io_uring for reads.** For a fully-resident table reads never touch a - **io_uring for reads.** For a fully-resident table reads never touch a
descriptor, so there is nothing to accelerate on the read path. This is a descriptor, so there is nothing to accelerate on the read path. This is a
write-durability optimization, full stop. **Note (2026-08-26):** principle 7's write-durability optimization, full stop. **Note (2026-08-26):** principle 7's
residency half was amended, so a table declaring `resident: index` residency half was amended, so a table declaring `resident: keys`
([databasev2 2](02-table-storage-modes.md)) *does* `pread` rows from the log — ([databasev2 2](02-table-storage-modes.md)) *does* `pread` rows from the log —
and accelerating that read path with io_uring becomes a real, separate and accelerating that read path with io_uring becomes a real, separate
question. It is not this iteration's, and it should not be folded in: this question. It is not this iteration's, and it should not be folded in: this

View file

@ -8,11 +8,12 @@ status: refine
# databasev2 7 — `WO_DATA=<path>.db`: the persistent store as one file # databasev2 7 — `WO_DATA=<path>.db`: the persistent store as one file
> **Moved 2026-08-26** from the language track, where this was iteration 33. > **Moved 2026-08-26** from the language track, where this was iteration 33.
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by > Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by
> the move; its dependencies are restated in that track index. > the move; its dependencies are restated in that track index.
> Format: `product/story-iteration-template`. Part of > Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). > [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md)
> — the track this iteration was authored in before the 2026-08-26 move.
> >
> **Inserted 2026-08-22** (developer ask: "can the persistent db be in > **Inserted 2026-08-22** (developer ask: "can the persistent db be in
> file.db form?"). The truth is already almost there: `WO_DATA=<dir>` > file.db form?"). The truth is already almost there: `WO_DATA=<dir>`

View file

@ -8,11 +8,12 @@ status: hold
# databasev2 8 — query grammar, driven by real embedded-DB corpora # databasev2 8 — query grammar, driven by real embedded-DB corpora
> **Moved 2026-08-26** from the language track, where this was iteration 27. > **Moved 2026-08-26** from the language track, where this was iteration 27.
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by > Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by
> the move; its dependencies are restated in that track index. > the move; its dependencies are restated in that track index.
> Format: `product/story-iteration-template`. Part of > Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). > [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md)
> — the track this iteration was authored in before the 2026-08-26 move.
> >
> **Inserted 2026-08-16.** A query-surface iteration in the 9b family: the > **Inserted 2026-08-16.** A query-surface iteration in the 9b family: the
> language-integrated query grows to cover the grammar that *real > language-integrated query grows to cover the grammar that *real

View file

@ -8,11 +8,12 @@ status: hold
# databasev2 9 — cross-program tables: attach to a running program's database # databasev2 9 — cross-program tables: attach to a running program's database
> **Moved 2026-08-26** from the language track, where this was iteration 20. > **Moved 2026-08-26** from the language track, where this was iteration 20.
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by > Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by
> the move; its dependencies are restated in that track index. > the move; its dependencies are restated in that track index.
> Format: `product/story-iteration-template`. Part of > Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). > [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md)
> — the track this iteration was authored in before the 2026-08-26 move.
> >
> **Inserted 2026-08-15**, hence `20`. It follows 9b because a program > **Inserted 2026-08-15**, hence `20`. It follows 9b because a program
> attaching to another's tables wants the same typed statements and queries > attaching to another's tables wants the same typed statements and queries

View file

@ -8,11 +8,12 @@ status: hold
# databasev2 10 — keypair authentication for cross-program attach # databasev2 10 — keypair authentication for cross-program attach
> **Moved 2026-08-26** from the language track, where this was iteration 21. > **Moved 2026-08-26** from the language track, where this was iteration 21.
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by > Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by
> the move; its dependencies are restated in that track index. > the move; its dependencies are restated in that track index.
> Format: `product/story-iteration-template`. Part of > Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). > [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md)
> — the track this iteration was authored in before the 2026-08-26 move.
> >
> **Inserted 2026-08-15.** Promotes iteration 20's identity fork (Info, > **Inserted 2026-08-15.** Promotes iteration 20's identity fork (Info,
> fork 3) to its own iteration: the name + unix-uid lean is the milestone > fork 3) to its own iteration: the name + unix-uid lean is the milestone

View file

@ -14,7 +14,7 @@
| Fork | Decision | | Fork | Decision |
| --- | --- | | --- | --- |
| One enum or two keys | **Two keys.** `durable:` and `resident:` answer two different developer questions ("do I need this after a restart?", "does it fit in RAM?"). One enum forces a name for each *combination*, which is what made a third value unreadable. | | One enum or two keys | **Two keys.** `durable:` and `resident:` answer two different developer questions ("do I need this after a restart?", "does it fit in RAM?"). One enum forces a name for each *combination*, which is what made a third value unreadable. |
| Mode vocabulary | **`resident: all \| index`** and **`durable: true \| false`**. No `cold`, `tiered`, `paged`, `mmap` or `buffer` in the grammar. | | Mode vocabulary | **`resident: all \| keys`** and **`durable: true \| false`**. No `cold`, `tiered`, `paged`, `mmap` or `buffer` in the grammar. |
| Optional or mandatory | **Optional, both default to today's behaviour** (`durable: true`, `resident: all`). All 28 existing declarations compile unchanged; no goldens reblessed. | | Optional or mandatory | **Optional, both default to today's behaviour** (`durable: true`, `resident: all`). All 28 existing declarations compile unchanged; no goldens reblessed. |
| Which storage architecture | **One engine, log-structured.** The WAL already holds every row; keep an in-RAM id→offset map and read rows back with `pread`. No second engine. | | Which storage architecture | **One engine, log-structured.** The WAL already holds every row; keep an in-RAM id→offset map and read rows back with `pread`. No second engine. |
| Row cache | **None in user space.** The kernel page cache is the hot copy — the repo's own stated position in `exploration/postgresql/buffer-and-checkpoint.md`: "`pread` against an fd that already has its page cached is a memcpy… the page cache is the one cache we want", and the reason the engine avoids `O_DIRECT`. | | Row cache | **None in user space.** The kernel page cache is the hot copy — the repo's own stated position in `exploration/postgresql/buffer-and-checkpoint.md`: "`pread` against an fd that already has its page cached is a memcpy… the page cache is the one cache we want", and the reason the engine avoids `O_DIRECT`. |
@ -67,19 +67,21 @@ grows two fields beside `table_name` and `indexes`.
| Argument | Values | Default | Meaning | | Argument | Values | Default | Meaning |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| `durable` | `true`, `false` | `true` | `false` skips the WAL append entirely: no record, no fsync, ack from RAM, table empty after restart. | | `durable` | `true`, `false` | `true` | `false` skips the WAL append entirely: no record, no fsync, ack from RAM, table empty after restart. |
| `resident` | `all`, `index` | `all` | `index` keeps the id map and every secondary index in RAM; rows are read from the log by offset. | | `resident` | `all`, `keys` | `all` | `keys` keeps the id map, every secondary index and every unique shadow in RAM; rows are read from the log by offset. |
`true`/`false` are already keyword tokens; `all`/`index` are parsed as the same `true`/`false` are already keyword tokens; `all`/`index` are parsed as the same
bare identifiers the `index:` argument's column list already accepts. No lexer bare identifiers the `index:` argument's column list already accepts. No lexer
change. change.
**One wart, surfaced rather than buried:** `resident: index` puts the word **Named `keys`, not `index`, on review (2026-08-26).** An earlier draft used
`index` in value position while `index:` is also a key, so the value `index`, which put `index` in value position while `index:` is also a
`@table(index: [customer], resident: index)` reads awkwardly on first key — `@table(index: [customer], resident: index)` read awkwardly (that line is
encounter. The parser distinguishes them structurally and there is no the rejected spelling, quoted). `keys` also
ambiguity, but a reviewer may prefer `resident: keys` or `resident: index_only`. puts both values on one axis: `all` and `keys` each answer "what row data stays
Flagged for the review of this document; the mechanism is unaffected either resident", where `all`/`index` mixed a quantity with a structure name. Neither
way. `all` nor `keys` is a keyword or a builtin (`key_at`/`val_at` exist; bare `keys`
does not). `resident: none` was considered and rejected as overclaiming — the
indexes are very much resident.
### The four combinations ### The four combinations
@ -87,12 +89,12 @@ way.
| --- | --- | --- | | --- | --- | --- |
| `true` | `all` | Today's behaviour. The default. Reads at memory speed. | | `true` | `all` | Today's behaviour. The default. Reads at memory speed. |
| `false` | `all` | Volatile scratch: sessions, rate-limit counters, idempotency keys. Skips the 66× fsync cost. What porch 1–3 need. | | `false` | `all` | Volatile scratch: sessions, rate-limit counters, idempotency keys. Skips the 66× fsync cost. What porch 1–3 need. |
| `true` | `index` | The 120 GB case. Rows in the log, indexes resident, `pread` on read. | | `true` | `keys` | The 120 GB case. Rows in the log, indexes resident, `pread` on read. |
| `false` | `index` | **Refused at compile time.** Rows would have nowhere to be read from. | | `false` | `keys` | **Refused at compile time.** Rows would have nowhere to be read from. |
### Read, write and recovery paths ### Read, write and recovery paths
- **Insert** — unchanged for `resident: all`. For `resident: index`: encode and - **Insert** — unchanged for `resident: all`. For `resident: keys`: encode and
append the record as today, then record id→offset in the resident map instead append the record as today, then record id→offset in the resident map instead
of retaining the row in a slab. The WAL append is already the durable write; of retaining the row in a slab. The WAL append is already the durable write;
this stops discarding its payload. this stops discarding its payload.
@ -136,7 +138,7 @@ only replayable.
- **Foreign-key restrict works unchanged.** It is a secondary-index probe, and - **Foreign-key restrict works unchanged.** It is a secondary-index probe, and
secondary indexes are resident. secondary indexes are resident.
- **`ref` navigation works unchanged**, at the cost of a `pread` per hop. - **`ref` navigation works unchanged**, at the cost of a `pread` per hop.
- **A `resident: all` table may hold a `ref` into a `resident: index` table** - **A `resident: all` table may hold a `ref` into a `resident: keys` table**
and vice versa — both are durable, so neither evaporates. This is the case and vice versa — both are durable, so neither evaporates. This is the case
that a `durable: false` table genuinely breaks, below. that a `durable: false` table genuinely breaks, below.
@ -145,7 +147,7 @@ only replayable.
Four refusals, each a catalogued `WO-E1xx` diagnostic added in the same change Four refusals, each a catalogued `WO-E1xx` diagnostic added in the same change
as the code — not afterwards: as the code — not afterwards:
1. `durable: false` with `resident: index` — the meaningless combination. 1. `durable: false` with `resident: keys` — the meaningless combination.
2. An unknown value for either argument, or either argument given twice. 2. An unknown value for either argument, or either argument given twice.
3. **A `durable: true` table holding a `ref` into a `durable: false` table.** 3. **A `durable: true` table holding a `ref` into a `durable: false` table.**
A persistent row cannot reference one that evaporates on restart; FK restrict A persistent row cannot reference one that evaporates on restart; FK restrict
@ -229,7 +231,7 @@ Acceptance is the story's Given/When/Then list; this is how each is exercised.
unchanged; no golden reblessed. unchanged; no golden reblessed.
- **Volatility** — a `durable: false` table produces no WAL growth (measured, - **Volatility** — a `durable: false` table produces no WAL growth (measured,
not asserted) and is empty after restart while durable siblings replay intact. not asserted) and is empty after restart while durable siblings replay intact.
- **Residency correctness** — a `resident: index` table larger than the - **Residency correctness** — a `resident: keys` table larger than the
configured budget returns every row correctly by id, byte-identical including configured budget returns every row correctly by id, byte-identical including
every heap-valued column, and scans in full. every heap-valued column, and scans in full.
- **Constraints across the boundary** — `@unique` refuses a duplicate whose - **Constraints across the boundary** — `@unique` refuses a duplicate whose
@ -240,8 +242,8 @@ Acceptance is the story's Given/When/Then list; this is how each is exercised.
- **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup; - **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup;
a budget breach names the table and the annotation. a budget breach names the table and the annotation.
- **Crash safety** — `kill -9` mid-append and mid-checkpoint on a - **Crash safety** — `kill -9` mid-append and mid-checkpoint on a
`resident: index` table; replay loses no acked write and no row appears twice. `resident: keys` table; replay loses no acked write and no row appears twice.
- **Performance** — new baseline rows for the `resident: index` read path with - **Performance** — new baseline rows for the `resident: keys` read path with
its amplification versus resident, published in `perf-targets.md` as a number its amplification versus resident, published in `perf-targets.md` as a number
a developer can plan around; and a regression check that resident tables did a developer can plan around; and a regression check that resident tables did
not move. not move.