diff --git a/docs/stories/databasev2/02-table-storage-modes.md b/docs/stories/databasev2/02-table-storage-modes.md index d46290c..df0cfcc 100644 --- a/docs/stories/databasev2/02-table-storage-modes.md +++ b/docs/stories/databasev2/02-table-storage-modes.md @@ -22,7 +22,7 @@ status: refine > **⚠ Superseded in part, 2026-08-26.** The brainstorm settled on a different > shape than this document describes: one log-structured engine where the WAL > *is* the row store, with residency declared per table (`resident: all` / -> `resident: index`) rather than a three-valued `mode:` enum including `cold`. +> `resident: keys`) rather than a three-valued `mode:` enum including `cold`. > Principle 7 was amended accordingly — the log is authoritative, residency is > the declaration. This file is rewritten once the grammar is approved; read the > [track index](00-story.md) and `docs/00-principles.md` §7 as current. diff --git a/docs/stories/databasev2/03-wal-checkpoint.md b/docs/stories/databasev2/03-wal-checkpoint.md index de775e1..2581214 100644 --- a/docs/stories/databasev2/03-wal-checkpoint.md +++ b/docs/stories/databasev2/03-wal-checkpoint.md @@ -9,11 +9,12 @@ chain: 6 # databasev2 3 — WAL checkpoint: disk space reclamation and bounded replay > **Moved 2026-08-26** from the language track, where this was iteration 32. -> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by +> Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by > the move; its dependencies are restated in that track index. > Format: `product/story-iteration-template`. Part of -> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). +> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md) +> — the track this iteration was authored in before the 2026-08-26 move. > > **Inserted 2026-08-21** (stage-3 guarantee refinement found the hole): > the WAL is append-only FOREVER — no checkpoint, no truncation exists diff --git a/docs/stories/databasev2/04-io-uring-commit.md b/docs/stories/databasev2/04-io-uring-commit.md index 5d8a819..28d7c31 100644 --- a/docs/stories/databasev2/04-io-uring-commit.md +++ b/docs/stories/databasev2/04-io-uring-commit.md @@ -9,11 +9,12 @@ chain: 5 # databasev2 4 — io_uring group-commit write path > **Moved 2026-08-26** from the language track, where this was iteration 23. -> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by +> Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by > the move; its dependencies are restated in that track index. > Format: `product/story-iteration-template`. Part of -> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). +> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md) +> — the track this iteration was authored in before the 2026-08-26 move. > > **Inserted 2026-08-15.** The write-path optimization, and deliberately the > LAST database performance iteration: it only earns its complexity once @@ -96,7 +97,7 @@ chain: 5 - **io_uring for reads.** For a fully-resident table reads never touch a descriptor, so there is nothing to accelerate on the read path. This is a write-durability optimization, full stop. **Note (2026-08-26):** principle 7's - residency half was amended, so a table declaring `resident: index` + residency half was amended, so a table declaring `resident: keys` ([databasev2 2](02-table-storage-modes.md)) *does* `pread` rows from the log — and accelerating that read path with io_uring becomes a real, separate question. It is not this iteration's, and it should not be folded in: this diff --git a/docs/stories/databasev2/07-single-file-db.md b/docs/stories/databasev2/07-single-file-db.md index 5e6a3b8..07854d8 100644 --- a/docs/stories/databasev2/07-single-file-db.md +++ b/docs/stories/databasev2/07-single-file-db.md @@ -8,11 +8,12 @@ status: refine # databasev2 7 — `WO_DATA=.db`: the persistent store as one file > **Moved 2026-08-26** from the language track, where this was iteration 33. -> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by +> Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by > the move; its dependencies are restated in that track index. > Format: `product/story-iteration-template`. Part of -> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). +> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md) +> — the track this iteration was authored in before the 2026-08-26 move. > > **Inserted 2026-08-22** (developer ask: "can the persistent db be in > file.db form?"). The truth is already almost there: `WO_DATA=` diff --git a/docs/stories/databasev2/08-query-grammar-corpus.md b/docs/stories/databasev2/08-query-grammar-corpus.md index 8113af6..4c5a35c 100644 --- a/docs/stories/databasev2/08-query-grammar-corpus.md +++ b/docs/stories/databasev2/08-query-grammar-corpus.md @@ -8,11 +8,12 @@ status: hold # databasev2 8 — query grammar, driven by real embedded-DB corpora > **Moved 2026-08-26** from the language track, where this was iteration 27. -> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by +> Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by > the move; its dependencies are restated in that track index. > Format: `product/story-iteration-template`. Part of -> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). +> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md) +> — the track this iteration was authored in before the 2026-08-26 move. > > **Inserted 2026-08-16.** A query-surface iteration in the 9b family: the > language-integrated query grows to cover the grammar that *real diff --git a/docs/stories/databasev2/09-cross-program-tables.md b/docs/stories/databasev2/09-cross-program-tables.md index 86b9bff..23add85 100644 --- a/docs/stories/databasev2/09-cross-program-tables.md +++ b/docs/stories/databasev2/09-cross-program-tables.md @@ -8,11 +8,12 @@ status: hold # databasev2 9 — cross-program tables: attach to a running program's database > **Moved 2026-08-26** from the language track, where this was iteration 20. -> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by +> Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by > the move; its dependencies are restated in that track index. > Format: `product/story-iteration-template`. Part of -> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). +> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md) +> — the track this iteration was authored in before the 2026-08-26 move. > > **Inserted 2026-08-15**, hence `20`. It follows 9b because a program > attaching to another's tables wants the same typed statements and queries diff --git a/docs/stories/databasev2/10-keypair-attach-auth.md b/docs/stories/databasev2/10-keypair-attach-auth.md index b3bdc78..fe2d960 100644 --- a/docs/stories/databasev2/10-keypair-attach-auth.md +++ b/docs/stories/databasev2/10-keypair-attach-auth.md @@ -8,11 +8,12 @@ status: hold # databasev2 10 — keypair authentication for cross-program attach > **Moved 2026-08-26** from the language track, where this was iteration 21. -> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by +> Part of [Story — databasev2: the database beyond RAM](00-story.md). Content unchanged by > the move; its dependencies are restated in that track index. > Format: `product/story-iteration-template`. Part of -> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md). +> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md) +> — the track this iteration was authored in before the 2026-08-26 move. > > **Inserted 2026-08-15.** Promotes iteration 20's identity fork (Info, > fork 3) to its own iteration: the name + unix-uid lean is the milestone diff --git a/docs/superpowers/specs/2026-08-26-table-residency-design.md b/docs/superpowers/specs/2026-08-26-table-residency-design.md index 48820d1..67f10a1 100644 --- a/docs/superpowers/specs/2026-08-26-table-residency-design.md +++ b/docs/superpowers/specs/2026-08-26-table-residency-design.md @@ -14,7 +14,7 @@ | Fork | Decision | | --- | --- | | One enum or two keys | **Two keys.** `durable:` and `resident:` answer two different developer questions ("do I need this after a restart?", "does it fit in RAM?"). One enum forces a name for each *combination*, which is what made a third value unreadable. | -| Mode vocabulary | **`resident: all \| index`** and **`durable: true \| false`**. No `cold`, `tiered`, `paged`, `mmap` or `buffer` in the grammar. | +| Mode vocabulary | **`resident: all \| keys`** and **`durable: true \| false`**. No `cold`, `tiered`, `paged`, `mmap` or `buffer` in the grammar. | | Optional or mandatory | **Optional, both default to today's behaviour** (`durable: true`, `resident: all`). All 28 existing declarations compile unchanged; no goldens reblessed. | | Which storage architecture | **One engine, log-structured.** The WAL already holds every row; keep an in-RAM id→offset map and read rows back with `pread`. No second engine. | | Row cache | **None in user space.** The kernel page cache is the hot copy — the repo's own stated position in `exploration/postgresql/buffer-and-checkpoint.md`: "`pread` against an fd that already has its page cached is a memcpy… the page cache is the one cache we want", and the reason the engine avoids `O_DIRECT`. | @@ -67,19 +67,21 @@ grows two fields beside `table_name` and `indexes`. | Argument | Values | Default | Meaning | | --- | --- | --- | --- | | `durable` | `true`, `false` | `true` | `false` skips the WAL append entirely: no record, no fsync, ack from RAM, table empty after restart. | -| `resident` | `all`, `index` | `all` | `index` keeps the id map and every secondary index in RAM; rows are read from the log by offset. | +| `resident` | `all`, `keys` | `all` | `keys` keeps the id map, every secondary index and every unique shadow in RAM; rows are read from the log by offset. | `true`/`false` are already keyword tokens; `all`/`index` are parsed as the same bare identifiers the `index:` argument's column list already accepts. No lexer change. -**One wart, surfaced rather than buried:** `resident: index` puts the word -`index` in value position while `index:` is also a key, so -`@table(index: [customer], resident: index)` reads awkwardly on first -encounter. The parser distinguishes them structurally and there is no -ambiguity, but a reviewer may prefer `resident: keys` or `resident: index_only`. -Flagged for the review of this document; the mechanism is unaffected either -way. +**Named `keys`, not `index`, on review (2026-08-26).** An earlier draft used +the value `index`, which put `index` in value position while `index:` is also a +key — `@table(index: [customer], resident: index)` read awkwardly (that line is +the rejected spelling, quoted). `keys` also +puts both values on one axis: `all` and `keys` each answer "what row data stays +resident", where `all`/`index` mixed a quantity with a structure name. Neither +`all` nor `keys` is a keyword or a builtin (`key_at`/`val_at` exist; bare `keys` +does not). `resident: none` was considered and rejected as overclaiming — the +indexes are very much resident. ### The four combinations @@ -87,12 +89,12 @@ way. | --- | --- | --- | | `true` | `all` | Today's behaviour. The default. Reads at memory speed. | | `false` | `all` | Volatile scratch: sessions, rate-limit counters, idempotency keys. Skips the 66× fsync cost. What porch 1–3 need. | -| `true` | `index` | The 120 GB case. Rows in the log, indexes resident, `pread` on read. | -| `false` | `index` | **Refused at compile time.** Rows would have nowhere to be read from. | +| `true` | `keys` | The 120 GB case. Rows in the log, indexes resident, `pread` on read. | +| `false` | `keys` | **Refused at compile time.** Rows would have nowhere to be read from. | ### Read, write and recovery paths -- **Insert** — unchanged for `resident: all`. For `resident: index`: encode and +- **Insert** — unchanged for `resident: all`. For `resident: keys`: encode and append the record as today, then record id→offset in the resident map instead of retaining the row in a slab. The WAL append is already the durable write; this stops discarding its payload. @@ -136,7 +138,7 @@ only replayable. - **Foreign-key restrict works unchanged.** It is a secondary-index probe, and secondary indexes are resident. - **`ref` navigation works unchanged**, at the cost of a `pread` per hop. -- **A `resident: all` table may hold a `ref` into a `resident: index` table** +- **A `resident: all` table may hold a `ref` into a `resident: keys` table** and vice versa — both are durable, so neither evaporates. This is the case that a `durable: false` table genuinely breaks, below. @@ -145,7 +147,7 @@ only replayable. Four refusals, each a catalogued `WO-E1xx` diagnostic added in the same change as the code — not afterwards: -1. `durable: false` with `resident: index` — the meaningless combination. +1. `durable: false` with `resident: keys` — the meaningless combination. 2. An unknown value for either argument, or either argument given twice. 3. **A `durable: true` table holding a `ref` into a `durable: false` table.** A persistent row cannot reference one that evaporates on restart; FK restrict @@ -229,7 +231,7 @@ Acceptance is the story's Given/When/Then list; this is how each is exercised. unchanged; no golden reblessed. - **Volatility** — a `durable: false` table produces no WAL growth (measured, not asserted) and is empty after restart while durable siblings replay intact. -- **Residency correctness** — a `resident: index` table larger than the +- **Residency correctness** — a `resident: keys` table larger than the configured budget returns every row correctly by id, byte-identical including every heap-valued column, and scans in full. - **Constraints across the boundary** — `@unique` refuses a duplicate whose @@ -240,8 +242,8 @@ Acceptance is the story's Given/When/Then list; this is how each is exercised. - **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup; a budget breach names the table and the annotation. - **Crash safety** — `kill -9` mid-append and mid-checkpoint on a - `resident: index` table; replay loses no acked write and no row appears twice. -- **Performance** — new baseline rows for the `resident: index` read path with + `resident: keys` table; replay loses no acked write and no row appears twice. +- **Performance** — new baseline rows for the `resident: keys` read path with its amplification versus resident, published in `perf-targets.md` as a number a developer can plan around; and a regression check that resident tables did not move.