docs(databasev2): rewrite iteration 2 to match what was designed and built
Flagged by the developer: the iteration still described the pre-brainstorm three-mode design behind a "superseded in part" banner while four tasks had landed against it. - iteration 2 rewritten around the shape as built: two keys (`durable: true|false`, `resident: all|keys`), not a `mode:` enum with `cold`. status refine -> in-progress - added a task-by-task progress table with commit hashes, and split the acceptance criteria into MET (each with how it was verified, not just that it passed — e.g. the goldens-unchanged claim is `git diff` over golden/ being empty after a WOC_BLESS run, since blessing rewrites all of them) and OUTSTANDING with the task that owns each - kept the history rather than deleting it: the three-mode replacement, the "one real rewrite" that was fiction, and the opposite half that turned out genuinely deep. An iteration file is where that record belongs - board row rewritten to agree; the track index's "the lever" section, its principle-7 paragraph and its sequence rationale all still taught the dead three-mode design ITERATION 6 IS NOW LARGELY SUPERSEDED, and bannered as such rather than quietly gutted. `resident: keys` is the ceiling-raiser and it lives in iteration 2 (tasks 5c/5d). More than relocated: 6's premise — a user-space resident working set with faulting and 5's eviction policy — was specifically REJECTED by the spec in favour of the kernel page cache, since a pread against a cached page is a memcpy. What may still be left is recorded honestly: revisit only with a measurement showing the page cache insufficient. Its fork list survives, especially "does the language surface the fault cost at the use site", which is still open and still the largest question about what writeonce is. The sequence rationale is amended too — it had 6 as the ceiling-raiser and 5 as a prerequisite on the critical path; neither holds. Docs only. linkcheck 0 broken / 0 anchors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
18a56f24ec
commit
51dd42f9db
4 changed files with 215 additions and 198 deletions
|
|
@ -620,7 +620,7 @@ the language arc as v1 history.
|
||||||
| # | Iteration | State |
|
| # | Iteration | State |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| 1 | [RAM ceiling: measure the breaking point](databasev2/01-ram-ceiling-measurement.md) | ⬜ **first, and startable today** — nobody here can say what happens at 90% RAM. Curve not cliff: swap onset, latency departure, the three exits (checked trap / swap thrash / OOM killer), and `kill -9` durability *at exhaustion*. Output is `perf-targets.md` + baseline rows, not prose |
|
| 1 | [RAM ceiling: measure the breaking point](databasev2/01-ram-ceiling-measurement.md) | ⬜ **first, and startable today** — nobody here can say what happens at 90% RAM. Curve not cliff: swap onset, latency departure, the three exits (checked trap / swap thrash / OOM killer), and `kill -9` durability *at exhaustion*. Output is `perf-targets.md` + baseline rows, not prose |
|
||||||
| 2 | [`@table` storage modes](databasev2/02-table-storage-modes.md) | ⬜ **the language enrichment** — `mode: ram \| durable \| cold` per table, replacing the global switch. `durable` defaults so nothing changes silently; the compiler refuses a `durable` row holding a `ref` into a `ram` table. `.wob` format change. Grammar is small (`Ast.table_cfg` gains a key); semantics are the iteration |
|
| 2 | [per-table storage: `durable` and `resident`](databasev2/02-table-storage-modes.md) | 🔄 **the language enrichment — the `durable` half is DONE and usable.** Two optional `@table` keys, `durable: true\|false` and `resident: all\|keys`, both defaulting to today's behaviour (all 28 existing declarations compile unchanged, no golden moved). Landed: the grammar, WO-E224 (a durable `ref` into a volatile table is refused), `.wob` v7 carrying both properties in spare `flags` bits, `durable: false` actually skipping the WAL (measured: 50 inserts → 1500 bytes durable, **0** volatile) with a mode-mismatch startup refusal, plus offset capture and read-a-row-from-an-offset. Outstanding: 5c/5d (the id→offset map and rewiring `wo_row_ptr`'s 11 call sites, slab scans and `@unique`/FK across the boundary — not yet written up), the two runtime refusals, and closeout. [spec](../superpowers/specs/2026-08-26-table-residency-design.md) · [plan](../superpowers/plans/2026-08-26-table-residency.md) |
|
||||||
| 3 | [WAL checkpoint](databasev2/03-wal-checkpoint.md) *(was 32)* | ⬜ snapshot + truncate: disk reclaimed, replay bounded |
|
| 3 | [WAL checkpoint](databasev2/03-wal-checkpoint.md) *(was 32)* | ⬜ snapshot + truncate: disk reclaimed, replay bounded |
|
||||||
| 4 | [io_uring group commit](databasev2/04-io-uring-commit.md) *(was 23)* | ⬜ close the 66× gap iteration 22 measured (durable 4.5k vs ram 297k inserts/s) |
|
| 4 | [io_uring group commit](databasev2/04-io-uring-commit.md) *(was 23)* | ⬜ close the 66× gap iteration 22 measured (durable 4.5k vs ram 297k inserts/s) |
|
||||||
| 5 | [Bounded tables and eviction](databasev2/05-bounded-tables-eviction.md) | ⬜ a declared capacity + refuse/evict/back-pressure, and a process-level pressure signal that sheds **before** the allocator or OS gets involved — turning the invisible failure into a managed one |
|
| 5 | [Bounded tables and eviction](databasev2/05-bounded-tables-eviction.md) | ⬜ a declared capacity + refuse/evict/back-pressure, and a process-level pressure signal that sheds **before** the allocator or OS gets involved — turning the invisible failure into a managed one |
|
||||||
|
|
|
||||||
|
|
@ -76,42 +76,49 @@ disposable*; an orders table wants *resident and durable*; an audit log wants
|
||||||
*durable and rarely read*. One global switch cannot express that, so it forces
|
*durable and rarely read*. One global switch cannot express that, so it forces
|
||||||
either "everything is precious" or "nothing is".
|
either "everything is precious" or "nothing is".
|
||||||
|
|
||||||
Extending `@table` with a storage mode moves the decision into the language,
|
Extending `@table` moves the decision into the language, where the compiler can
|
||||||
where the compiler can act on it:
|
act on it. **Two keys, not one enum** — the developer is answering two
|
||||||
|
independent questions, and an enum would need a name for every combination:
|
||||||
|
|
||||||
- **`ram`** — resident, never WAL-logged, gone on restart. The compiler knows
|
- **`durable: true | false`** (default `true`). `false` skips the WAL append
|
||||||
no durability code is needed; the engine knows these rows are the first
|
entirely: no record, no fsync, ack from RAM, table empty after restart. The
|
||||||
candidates to shed under pressure; and — the part that matters — a program
|
compiler can then refuse a program that stores a durable `ref` into such a
|
||||||
that expects a `ram` table to survive a restart is now stating something the
|
table, because that id would dangle across a restart (WO-E224).
|
||||||
compiler can refuse.
|
- **`resident: all | keys`** (default `all`). `keys` keeps the id map, the
|
||||||
- **`durable`** — today's behaviour: resident and WAL-logged, ack after fsync.
|
secondary indexes and the unique shadows resident and reads rows back from
|
||||||
- **`cold`** — durable, and *not* required to be resident. This is the mode that
|
the log by offset. This is the key that raises the ceiling — and the
|
||||||
actually raises the ceiling, and it is the one with real design work behind it
|
arithmetic is why it works: 240M rows × 16 B of index ≈ 3.8 GB resident for a
|
||||||
(iteration [6](06-cold-tiering.md)).
|
120 GB table.
|
||||||
|
|
||||||
The grammar change is small and the surface is already the right shape:
|
`durable: false` with `resident: keys` is refused: rows would be neither logged
|
||||||
`Ast.table_cfg` is `{ table_name; indexes }`, the parser's argument match
|
nor resident, so there would be nowhere to read them from.
|
||||||
already rejects unknown keys with a catalogued diagnostic
|
|
||||||
(`unknown @table argument ... (supported: name, index)`), and adding one more
|
|
||||||
key follows the path `index` already cut. The *semantics* are the work, not the
|
|
||||||
syntax — which is exactly why it gets its own iteration
|
|
||||||
([2](02-table-storage-modes.md)) and why it comes after the measurement.
|
|
||||||
|
|
||||||
This is also the honest answer to "does this break principle 7?" It does not.
|
The grammar change was small, as predicted — `Ast.table_cfg` gained two fields
|
||||||
RAM stays authoritative **for the tables that say so**. `cold` is a declared
|
and the parser's argument match two arms. The *semantics* were the work, which
|
||||||
exception a developer opts into per table, with the trade written at the
|
is why iteration 2 is 7 tasks rather than one.
|
||||||
declaration site rather than buried in an operations runbook.
|
|
||||||
|
**Does this break principle 7?** It amends it, deliberately, and the amendment
|
||||||
|
is applied: the **log** is authoritative and residency is a declared per-table
|
||||||
|
policy. Durability is untouched and unconditional — ack after fsync, replay
|
||||||
|
whole-or-nothing, torn tails dropped by CRC. What stays rejected is a *second*
|
||||||
|
engine: a paged B-tree with its own buffer pool. Reading rows from the log we
|
||||||
|
already write is not that.
|
||||||
|
|
||||||
|
An earlier draft of this section proposed a three-valued `mode:` enum including
|
||||||
|
`cold`. That name conflated durability with residency and could not be defined
|
||||||
|
before its mechanism existed; the history is in
|
||||||
|
[iteration 2](02-table-storage-modes.md).
|
||||||
|
|
||||||
## The sequence
|
## The sequence
|
||||||
|
|
||||||
| # | Iteration | Delivers | Needs |
|
| # | Iteration | Delivers | Needs |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| 1 | [RAM ceiling: measure the breaking point](01-ram-ceiling-measurement.md) | what actually happens from 50% RAM to OOM — swap onset, latency cliff, trap behaviour, `kill -9` survival | nothing; extends iteration 22's harness |
|
| 1 | [RAM ceiling: measure the breaking point](01-ram-ceiling-measurement.md) | what actually happens from 50% RAM to OOM — swap onset, latency cliff, trap behaviour, `kill -9` survival | nothing; extends iteration 22's harness |
|
||||||
| 2 | [`@table` storage modes](02-table-storage-modes.md) | the grammar: `mode: ram \| durable \| cold`, per table, replacing the global `WO_DATA` all-or-nothing | 1 for its defaults |
|
| 2 | [per-table storage](02-table-storage-modes.md) | the grammar: `durable: true\|false` and `resident: all\|keys`, per table, replacing the global `WO_DATA` all-or-nothing. **In progress — the `durable` half is done** | 1 for the budget default |
|
||||||
| 3 | [WAL checkpoint](03-wal-checkpoint.md) *(was language 32)* | snapshot + truncate: disk reclaimed, replay bounded | 4 composes |
|
| 3 | [WAL checkpoint](03-wal-checkpoint.md) *(was language 32)* | snapshot + truncate: disk reclaimed, replay bounded | 4 composes |
|
||||||
| 4 | [io_uring group commit](04-io-uring-commit.md) *(was language 23)* | close the 66× durable/RAM write gap (4.5k vs 297k inserts/s) | the arc (landed) |
|
| 4 | [io_uring group commit](04-io-uring-commit.md) *(was language 23)* | close the 66× durable/RAM write gap (4.5k vs 297k inserts/s) | the arc (landed) |
|
||||||
| 5 | [Bounded tables and eviction](05-bounded-tables-eviction.md) | a capacity a `ram` table may not exceed, and what happens when it does | 2 |
|
| 5 | [Bounded tables and eviction](05-bounded-tables-eviction.md) | a capacity a `ram` table may not exceed, and what happens when it does | 2 |
|
||||||
| 6 | [Cold tiering](06-cold-tiering.md) | rows that leave RAM and come back — the iteration that raises the ceiling | 2, 3, 5 |
|
| 6 | [Cold tiering](06-cold-tiering.md) | ⚠ **largely superseded by 2** — `resident: keys` is the ceiling-raiser. Its premise (a user-space resident working set) was rejected in favour of the kernel page cache. Revisit only with a measurement showing the page cache insufficient | — |
|
||||||
| 7 | [Single-file store](07-single-file-db.md) *(was language 33)* | `WO_DATA=<path>.db` — a file path IS the store | independent |
|
| 7 | [Single-file store](07-single-file-db.md) *(was language 33)* | `WO_DATA=<path>.db` — a file path IS the store | independent |
|
||||||
| 8 | [Query grammar from corpora](08-query-grammar-corpus.md) *(was language 27)* | whole-query `count`, `exists` | independent |
|
| 8 | [Query grammar from corpora](08-query-grammar-corpus.md) *(was language 27)* | whole-query `count`, `exists` | independent |
|
||||||
| 9 | [Cross-program tables](09-cross-program-tables.md) *(was language 20)* | attach to a running program's database over local IPC | independent |
|
| 9 | [Cross-program tables](09-cross-program-tables.md) *(was language 20)* | attach to a running program's database over local IPC | independent |
|
||||||
|
|
@ -125,11 +132,15 @@ declaration site rather than buried in an operations runbook.
|
||||||
9 ──▶ 10
|
9 ──▶ 10
|
||||||
```
|
```
|
||||||
|
|
||||||
Order rationale: **1 before 2** because a mode's default should follow from a
|
Order rationale: **1 before 2** because the budget default should follow from a
|
||||||
measurement, not a guess. **3 and 4 before 6** because tiering onto a log that
|
measurement, not a guess. **3 and 4 matter to 2** for the same reason tiering
|
||||||
never truncates would make the disk problem worse, not better. **5 before 6**
|
onto a never-truncating log would have: `resident: keys` rebuilds its offset map
|
||||||
because eviction from a bounded resident table is the simpler half of the same
|
by scanning the whole log at boot until 3's snapshot persists it.
|
||||||
mechanism, and getting the policy right there de-risks the hard half.
|
|
||||||
|
Amended 2026-08-27: the original rationale sequenced **6** as the ceiling-raiser
|
||||||
|
after 3, 4 and 5. `resident: keys` took that role into iteration 2, so 6 is
|
||||||
|
largely superseded and 5 is no longer a prerequisite for anything on the
|
||||||
|
critical path.
|
||||||
|
|
||||||
## What this track does NOT own
|
## What this track does NOT own
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,196 +1,180 @@
|
||||||
---
|
---
|
||||||
track: databasev2
|
track: databasev2
|
||||||
iteration: "2"
|
iteration: "2"
|
||||||
status: refine
|
status: in-progress
|
||||||
---
|
---
|
||||||
|
|
||||||
# databasev2 2 — `@table` storage modes: durability becomes a language decision
|
# databasev2 2 — per-table storage: `durable` and `resident`
|
||||||
|
|
||||||
> Part of [Story — databasev2: the database beyond RAM](00-story.md).
|
> Part of [Story — databasev2: the database beyond RAM](00-story.md).
|
||||||
> Needs [1](01-ram-ceiling-measurement.md) — a mode's default should follow from
|
> Spec: [`2026-08-26-table-residency-design.md`](../../superpowers/specs/2026-08-26-table-residency-design.md)
|
||||||
> a measurement, not a preference.
|
> · plan: [`2026-08-26-table-residency.md`](../../superpowers/plans/2026-08-26-table-residency.md).
|
||||||
>
|
>
|
||||||
> **The developer's ask, and the language enrichment this track exists for.**
|
> **The language enrichment this track exists for.** Before this, durability was
|
||||||
> Today durability is one environment variable for a whole process: `WO_DATA` is
|
> one environment variable for a whole process: `WO_DATA` set and every `@table`
|
||||||
> set and every `@table` is WAL-logged, or it is not and none are
|
> is WAL-logged, or unset and none are (`runtime/src/main.c`, and `db.c` guards
|
||||||
> (`runtime/src/main.c`). Real applications are not uniform. A session table, a
|
> each append on the WAL pointer). Real applications are not uniform — a session
|
||||||
> rate-limit counter and a page cache are resident and disposable; an orders
|
> table and a rate-limit counter are disposable, an orders table is precious, a
|
||||||
> table is resident and precious; an audit log is precious and rarely read. One
|
> 120 GB audit table does not fit in RAM at all. One global switch forces
|
||||||
> global switch forces "everything is precious" or "nothing is", and the
|
> "everything is precious" or "nothing is", and the developer pays for the wrong
|
||||||
> developer pays for the wrong one either way.
|
> one either way.
|
||||||
|
>
|
||||||
|
> **Rewritten 2026-08-27** to match what was designed and built. Two earlier
|
||||||
|
> drafts of this file described a three-valued `mode:` enum including `cold`;
|
||||||
|
> that design was replaced during the brainstorm and the history is at the
|
||||||
|
> bottom.
|
||||||
|
|
||||||
> **⚠ Superseded in part, 2026-08-26.** The brainstorm settled on a different
|
## The design, as built
|
||||||
> shape than this document describes: one log-structured engine where the WAL
|
|
||||||
> *is* the row store, with residency declared per table (`resident: all` /
|
|
||||||
> `resident: keys`) rather than a three-valued `mode:` enum including `cold`.
|
|
||||||
> Principle 7 was amended accordingly — the log is authoritative, residency is
|
|
||||||
> the declaration. This file is rewritten once the grammar is approved; read the
|
|
||||||
> [track index](00-story.md) and `docs/00-principles.md` §7 as current.
|
|
||||||
|
|
||||||
## Goals
|
Two optional `@table` arguments, because the developer is answering two
|
||||||
|
independent questions — *do I need this after a restart?* and *does it fit in
|
||||||
|
RAM?* A single enum would have forced a name for every combination, which is
|
||||||
|
what made the third value unwriteable before its mechanism existed.
|
||||||
|
|
||||||
- **Move the storage decision to the declaration site.** `@table(mode: ...)`,
|
| Argument | Values | Default | Meaning |
|
||||||
chosen per table, in the source, where the person who knows what the data is
|
| --- | --- | --- | --- |
|
||||||
worth is already writing. An operations runbook is the wrong place for a fact
|
| `durable` | `true`, `false` | `true` | `false` skips the WAL append entirely: no record, no fsync, ack from RAM, table empty after restart |
|
||||||
the compiler could hold.
|
| `resident` | `all`, `keys` | `all` | `keys` keeps the id map, secondary indexes and unique shadows resident; rows are read back from the log by offset |
|
||||||
- **Three modes, each earning its existence.** `ram` — resident, never logged,
|
|
||||||
gone on restart. `durable` — today's behaviour, resident and ack-after-fsync,
|
|
||||||
and the **default** so every existing program is byte-identical. `cold` —
|
|
||||||
durable and not required to be resident, declared here and *implemented* in
|
|
||||||
[6](06-cold-tiering.md), because a mode with no engine behind it is a promise.
|
|
||||||
- **Make the compiler enforce what the mode means.** This is the part that makes
|
|
||||||
it a language feature rather than a config key. A program that inserts into a
|
|
||||||
`ram` table and expects the row after a restart is stating a contradiction, and
|
|
||||||
the compiler is the right place to say so — at minimum for the cases it can
|
|
||||||
see statically, with the diagnostic catalogued like every other.
|
|
||||||
- **Let the engine act on it.** `ram` tables skip the WAL write entirely, which
|
|
||||||
is not merely a saving — it is the 66× gap iteration 22 measured (4.5k durable
|
|
||||||
vs 297k RAM inserts/s) becoming available per table instead of per process.
|
|
||||||
They also become the first candidates to shed under pressure, which is what
|
|
||||||
[5](05-bounded-tables-eviction.md) builds on.
|
|
||||||
- **Keep principle 7 intact and say why.** RAM stays authoritative for every
|
|
||||||
table that says so. `cold` is a declared, per-table exception a developer opts
|
|
||||||
into with the trade visible at the declaration.
|
|
||||||
|
|
||||||
## Phases
|
Both default to the pre-existing behaviour, which is why all 28 `@table`
|
||||||
|
declarations in the repository compiled unchanged and no golden moved.
|
||||||
|
`durable: false` with `resident: keys` is refused — rows would be neither
|
||||||
|
logged nor resident, so there would be nowhere to read them from.
|
||||||
|
|
||||||
### Phase A — the grammar
|
The engine stays **one log-structured store**. The WAL already held every row;
|
||||||
|
this iteration stops discarding the payload. No second engine, no user-space row
|
||||||
|
cache — the kernel page cache is the hot copy, which is the position
|
||||||
|
`exploration/postgresql/buffer-and-checkpoint.md` already argued and the reason
|
||||||
|
the engine avoids `O_DIRECT`.
|
||||||
|
|
||||||
- Extend the `@table` argument parser with `mode:`. The path is already cut:
|
Principle 7 was amended for this: the log is authoritative, residency is a
|
||||||
the argument loop matches `name` and `index` and rejects anything else with a
|
declared per-table policy. Durability is untouched and unconditional.
|
||||||
catalogued diagnostic (`unknown @table argument ... (supported: name, index)`),
|
|
||||||
so this is one more arm plus an updated message.
|
|
||||||
- Extend `Ast.table_cfg` — today `{ table_name; indexes }` — with the mode, and
|
|
||||||
give it the default so every existing `@table` keeps its meaning.
|
|
||||||
- Reject the incoherent cases at parse time: `mode` given twice, an unknown mode
|
|
||||||
name. Both belong in the same diagnostic family as the existing `@table`
|
|
||||||
errors, and both go in the error catalog **in this change**, not later — that
|
|
||||||
catalog went stale once by exactly that omission.
|
|
||||||
- Verify: golden AST fixtures for each mode; `woc-test` green; every existing
|
|
||||||
`@table` in every sample parses unchanged.
|
|
||||||
|
|
||||||
### Phase B — the mode reaches the image and the engine
|
## Progress
|
||||||
|
|
||||||
- Carry the mode through the class descriptor into the `.wob` image so the
|
| # | Task | State |
|
||||||
runtime knows it without re-deriving anything. This is a format change, so it
|
| --- | --- | --- |
|
||||||
moves `WOB_VERSION` and the format contract in the same commit as the code.
|
| 1 | grammar: both arguments, defaults preserve behaviour | ✅ `69b7ce2` |
|
||||||
- The engine consults it at the one choke point that already exists: the
|
| 2 | WO-E224: refuse a durable `ref` into a volatile table | ✅ `753e6c4` |
|
||||||
`INDEX HOOK` / WAL staging site in `wo_row_insert` / `wo_row_remove`, which
|
| 3 | `.wob` v7: the class descriptor carries both properties | ✅ `7e68c99` |
|
||||||
`database/src/CODE-LOGIC.md` names as the only place storage may be mutated.
|
| 4 | `durable: false` skips the WAL append and replay | ✅ `dd67e31` |
|
||||||
A `ram` table stages nothing.
|
| 5a | `wo_wal_next_offset` — exact record offsets | ✅ `ac7d8af` |
|
||||||
- Replay must skip records for tables that are now `ram` — a WAL written when a
|
| 5b | `wo_wal_read_row_at` — a row from a log offset | ✅ `d0c370c` |
|
||||||
table was `durable` and replayed after the source changed is a real
|
| 5c | the id→offset map + drop-payload-keep-index | ⬜ **not written up** |
|
||||||
migration case, and silently resurrecting rows into a `ram` table would be
|
| 5d | rewiring `wo_row_ptr`'s call sites, slab scans, `@unique`/FK across the boundary | ⬜ not written up |
|
||||||
worse than refusing.
|
| 6 | the two runtime refusals (no-`WO_DATA`, the byte budget) | ⬜ |
|
||||||
- Verify: a `ram` table's inserts produce no WAL growth (measured, not assumed);
|
| 7 | measure, gate, document, close out | ⬜ |
|
||||||
a `durable` table is byte-identical to today; a mode change across a restart
|
|
||||||
is handled explicitly rather than by accident.
|
|
||||||
|
|
||||||
### Phase C — the compiler's enforcement
|
**The `durable` half is complete and usable.** A volatile table is a full table
|
||||||
|
in-process — same indexes, same `@unique`, same FK restrict, same query surface
|
||||||
|
— and is simply empty after a restart. That is what
|
||||||
|
[porch 1–3](../porch/01-store-backed-middleware.md) need for sessions,
|
||||||
|
rate-limit counters and idempotency keys.
|
||||||
|
|
||||||
- Decide how far static checking goes (fork 2) and implement that much. The
|
**The `resident: keys` half has its read path but no storage behind it.**
|
||||||
floor: `WO_DATA` set with every table `ram` is a program that asked for a data
|
Offsets can be captured and rows can be read back from them; nothing yet stores
|
||||||
directory it will never write to — worth a warning at least.
|
a table that way.
|
||||||
- The interesting case is a `ram` table participating in a `ref`/`backlink`
|
|
||||||
relation with a `durable` one. A durable row holding a foreign key into a
|
|
||||||
table that evaporates on restart is a dangling reference by construction, and
|
|
||||||
FK-restrict cannot save it. This is the check most worth having, and it is
|
|
||||||
statically visible from the class table.
|
|
||||||
- Verify: corpus `compile-fail` fixtures for each refusal; each carries the
|
|
||||||
exact `WO-E###` the catalog now documents.
|
|
||||||
|
|
||||||
### Phase D — prove it on a real workload
|
|
||||||
|
|
||||||
- Give `docs/examples/employee` or the `db-bench` sample a mixed schema — at
|
|
||||||
least one `ram` table and one `durable` — and gate the distinction: after a
|
|
||||||
restart, the durable rows are present and the ram rows are gone. That single
|
|
||||||
assertion is the whole feature.
|
|
||||||
- Extend the durability legs of `just db-bench` so the per-table write-path
|
|
||||||
saving appears as a baseline number, not a claim.
|
|
||||||
- Verify: `just employee`, `just db-actor`, `just db-bench`, `oop-accept` green.
|
|
||||||
|
|
||||||
### Phase E — document the contract
|
|
||||||
|
|
||||||
- The `@table` mode surface in the language-surface guide and the db-binding
|
|
||||||
contract; the new diagnostics in the error catalog; the mode's effect on
|
|
||||||
replay in `database/src/CODE-LOGIC.md`.
|
|
||||||
- Verify: `just linkcheck` clean; the language-surface guide's `@table` row
|
|
||||||
matches what the parser actually accepts.
|
|
||||||
|
|
||||||
## Acceptance Criteria
|
## Acceptance Criteria
|
||||||
|
|
||||||
- **Given** every existing `@table` declaration in the repository, **when** it is
|
Met:
|
||||||
compiled after this change, **then** behaviour is byte-identical — `durable`
|
|
||||||
is the default and nothing opts in silently.
|
- **Given** every existing `@table` declaration, **when** compiled, **then**
|
||||||
- **Given** a table declared `mode: ram`, **when** rows are inserted with
|
behaviour is byte-identical. ✅ verified as `git diff` over
|
||||||
`WO_DATA` set, **then** the WAL does not grow, and after a restart the table is
|
`compiler/test/golden/` being empty after a `WOC_BLESS` run — a green test
|
||||||
empty while `durable` tables in the same program replay intact.
|
run alone proves nothing, since blessing rewrites every golden.
|
||||||
- **Given** `mode: ram` and a measured insert workload, **when** it runs against
|
- **Given** `durable: false` with `WO_DATA` set, **when** rows are inserted,
|
||||||
the same shape as a `durable` table, **then** the write-path saving is visible
|
**then** the WAL does not grow and the table is empty after a restart while
|
||||||
in `bench/baseline.json` — the per-table half of iteration 22's 66× gap.
|
durable siblings replay. ✅ measured: 50 inserts wrote 1500 bytes durable and
|
||||||
- **Given** an unknown mode name or `mode:` given twice, **when** it is compiled,
|
**0** volatile. Measured against the file's non-zero prefix, because the file
|
||||||
**then** it fails with the catalogued diagnostic naming the legal modes.
|
is `fallocate`'d to 1 MiB and its size proves nothing.
|
||||||
- **Given** a `durable` table holding a `ref` into a `ram` table, **when** it is
|
- **Given** an unknown value, a repeated argument, a retired design word, or the
|
||||||
compiled, **then** the compiler refuses (or warns, per fork 2) — a persistent
|
refused combination, **when** compiled, **then** WO-E102 with a message
|
||||||
row cannot reference one that evaporates.
|
naming what to write instead. ✅
|
||||||
- **Given** a WAL containing records for a table whose source now says `ram`,
|
- **Given** a `durable` table holding a `ref` into a volatile one, **when**
|
||||||
**when** the program starts, **then** the situation is handled explicitly
|
compiled, **then** WO-E224 naming both classes and both escapes. ✅ The
|
||||||
(refuse, or skip and report) and never by silently loading rows into a table
|
reverse direction and every `backlink` shape stay legal, pinned by a run
|
||||||
declared not to have any.
|
fixture so the check cannot grow over-broad.
|
||||||
- **Given** the `.wob` format change, **when** an image from the previous version
|
- **Given** a WAL holding records for a class the source now declares volatile,
|
||||||
is loaded, **then** the loader refuses it clearly on the version rather than
|
**when** the program starts, **then** it refuses, exits 2, names the class,
|
||||||
misreading a descriptor.
|
and is **not** reported as corruption. ✅
|
||||||
|
- **Given** a v6 image, **when** loaded, **then** refused on version rather
|
||||||
|
than misread. ✅
|
||||||
|
|
||||||
|
Outstanding:
|
||||||
|
|
||||||
|
- **Given** a `resident: keys` table larger than any plausible resident budget,
|
||||||
|
**when** rows are read by id and scanned, **then** every row is byte-identical
|
||||||
|
including heap-valued columns. *(needs 5c/5d)*
|
||||||
|
- **Given** `@unique` on a `resident: keys` table, **when** a duplicate arrives
|
||||||
|
whose conflicting row is not resident, **then** it is refused. *(5d — the
|
||||||
|
correctness core; a constraint that silently checks only resident rows must
|
||||||
|
never ship)*
|
||||||
|
- **Given** `durable: true` and no `WO_DATA`, **when** the program starts,
|
||||||
|
**then** it refuses. *(task 6 — today this combination silently discards
|
||||||
|
every write)*
|
||||||
|
- **Given** the resident footprint crossing the budget, **when** it does,
|
||||||
|
**then** a refusal naming the table and the annotation. *(task 6)*
|
||||||
|
- **Given** the `resident: all` read baseline, **when** re-measured, **then**
|
||||||
|
inside tolerance — no cost for a feature not used. *(task 7)*
|
||||||
|
|
||||||
## Out Of Scope
|
## Out Of Scope
|
||||||
|
|
||||||
- **Implementing `cold`.** Declared here so the mode set is settled and the
|
- **Checkpoint and compaction** — [3](03-wal-checkpoint.md). Boot rebuilds the
|
||||||
format carries it; the engine behaviour is [6](06-cold-tiering.md). Until then
|
offset map by scanning the log until that lands, which is O(all history);
|
||||||
a `cold` declaration must be refused rather than silently treated as
|
3's snapshot should persist the map.
|
||||||
`durable` — accepting a mode that does nothing is how a feature becomes a lie.
|
- **Eviction and a resident row cache** — [5](05-bounded-tables-eviction.md).
|
||||||
- **Per-table capacity limits and eviction** — [5](05-bounded-tables-eviction.md).
|
This iteration's tables are either fully resident or keys-only.
|
||||||
This iteration says what a table *is*; that one says how much of it there may
|
- **io_uring on the read path** — a real question that only exists after this;
|
||||||
be.
|
noted in [4](04-io-uring-commit.md), deliberately not folded in.
|
||||||
- **`@table` feature flags and `transaction { }`** — language
|
- **`transaction { }` and `@table` feature flags** — language
|
||||||
[iteration 18](../language-runtime-database/18-memory-db-features.md), whose
|
[iteration 18](../language-runtime-database/18-memory-db-features.md),
|
||||||
spec is approved and deliberately left whole.
|
approved spec, left whole.
|
||||||
- **Per-table WAL files.** One log, one writer, shard 0 — the invariant stage 3
|
- **Per-shard residency for volatile tables** — a volatile table has no WAL, so
|
||||||
established and [7](07-single-file-db.md) depends on. Modes decide *whether* a
|
it arguably need not live on the owner shard at all. Faster, and a different
|
||||||
table logs, never *where*.
|
consistency story. Recorded as a candidate, not decided.
|
||||||
- **Migrating an existing dataset between modes.** A schema-change story, and
|
- **Converting an existing dataset between settings.** Refuse on mismatch, do
|
||||||
the repo already records destructive migrations as a recorded future.
|
not convert — implemented in task 4.
|
||||||
- **Encryption at rest, compression of the WAL.** Neither has a consumer.
|
|
||||||
|
|
||||||
## Info
|
## Info — the forks, settled
|
||||||
|
|
||||||
The grammar surface this touches, read from the source: the parser's `@table`
|
1. **Two keys, not one enum.** An enum needs a name per *combination*, and the
|
||||||
argument loop and its `unknown @table argument` failure; `Ast.table_cfg` as
|
brainstorm demonstrated the third name is unwriteable before its mechanism is
|
||||||
`{ table_name : string option; indexes : string list list }`; and the class
|
decided.
|
||||||
descriptor in `runtime/src/wob.h` that the loader validates. Adding a key is
|
2. **`keys`, not `index`.** `index:` is already an argument key, so
|
||||||
genuinely small — the semantics are the iteration.
|
`@table(index: [c], resident: index)` read badly. `all`/`keys` also put both
|
||||||
|
values on one axis — what row data stays resident. `resident: none` was
|
||||||
|
rejected as overclaiming, since the indexes are very much resident.
|
||||||
|
3. **Optional with `durable` defaulting true**, not mandatory. Mandatory would
|
||||||
|
have touched 28 declarations, 13 corpus fixtures and 3 goldens; the README
|
||||||
|
already says nothing is API-stable, so making it mandatory at 1.0 stays
|
||||||
|
available.
|
||||||
|
4. **`@unique` on `resident: keys` is allowed**, with its index unconditionally
|
||||||
|
resident. Roughly doubles the resident index; stated at the declaration so
|
||||||
|
the cost is visible.
|
||||||
|
5. **The budget is bytes, not rows** — a text-heavy row and an Int-only row
|
||||||
|
differ by an order of magnitude, so a row count cannot bound RAM.
|
||||||
|
|
||||||
Forks the spec must settle:
|
## History — two corrections worth keeping
|
||||||
|
|
||||||
1. **What are the modes called?** `ram` / `durable` / `cold` is descriptive of
|
**The three-mode design was replaced.** Earlier drafts had
|
||||||
mechanism. `scratch` / `persistent` / `archived` is descriptive of intent and
|
`mode: ram | durable | cold`. `cold` conflated two independent properties and
|
||||||
is what a developer reasons about. The names are the API and are hard to
|
could not be named honestly before its mechanism existed, and the developer's
|
||||||
change later; leaning the intent-shaped set for the first two if a
|
120 GB-on-32 GB case showed the real axis was residency. Replaced by two keys,
|
||||||
short-enough pair can be found, since a developer choosing a mode is thinking
|
and principle 7 amended rather than worked around.
|
||||||
about what the data is *for*, not about where it sits.
|
|
||||||
2. **How hard does the compiler push?** Three levels: warn on the suspicious
|
**The "one real rewrite" was fiction.** The spec claimed the on-disk record was
|
||||||
cases; refuse the provably-broken ones (a `durable`→`ram` `ref`); or a full
|
pointer-bearing and that re-encoding it was this iteration's substantive
|
||||||
dataflow check that an insert into a `ram` table is never expected to persist.
|
engineering. That came from reading `table.c`'s `db_val_encode` — which builds
|
||||||
The third is not statically decidable in general. Leaning: refuse the
|
the *in-memory slot* — and inferring the file format from it. `wal.c`'s
|
||||||
relation case (provable, high value), warn on the `WO_DATA`-with-no-durable-
|
`enc_val` has been flat since iteration 9. The task was deleted, not reduced.
|
||||||
table case, and stop there.
|
|
||||||
3. **What is the default, and does it depend on iteration 1?** `durable` keeps
|
**The opposite half then turned out to be genuinely deep.** With the format
|
||||||
every existing program identical, which is nearly decisive. But if iteration
|
fine, the plan's storage steps still read as plumbing. Measured instead:
|
||||||
1's numbers show the WAL write dominating a workload nobody wanted durable,
|
`wo_row_ptr` returns a `db_row *` into a slab and has 11 call sites, `table.c`
|
||||||
there is an argument for making the choice mandatory — no default, every
|
has 37 slab references, `db.c:105-181` walks slabs for scans, `enc_val`
|
||||||
`@table` states its mode. That is a bigger source change and a better
|
serialises *from* the slab, and **no operation exists that drops a row's payload
|
||||||
language; the fork is whether the churn is worth it now or at 1.0.
|
while keeping its index entries**. Hence the 5a–5d split. 5c and 5d need their
|
||||||
4. **Does `mode: ram` imply anything about the actor/DB-actor path?** Stage 3
|
own write-ups, and the two open design questions for 5c are whether the id hash
|
||||||
marshals worker-shard statements to shard 0 because the owner shard holds the
|
stores offsets in place of slot indices or gains a parallel map, and what the
|
||||||
store and the WAL. A `ram` table has no WAL — so does it still need to live on
|
new operation does about the unique shadows, which currently point at slots.
|
||||||
the owner? A per-shard `ram` table would be dramatically faster and a
|
|
||||||
different consistency story. Tempting, out of scope here, and worth recording
|
|
||||||
as a candidate rather than deciding in passing.
|
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,28 @@ status: refine
|
||||||
> [3](03-wal-checkpoint.md) so the log this builds on does not grow forever,
|
> [3](03-wal-checkpoint.md) so the log this builds on does not grow forever,
|
||||||
> and [5](05-bounded-tables-eviction.md) for the policy machinery.
|
> and [5](05-bounded-tables-eviction.md) for the policy machinery.
|
||||||
>
|
>
|
||||||
|
> **⚠ LARGELY SUPERSEDED 2026-08-27 by [iteration 2](02-table-storage-modes.md).**
|
||||||
|
> This iteration was written to implement a `cold` mode. That mode no longer
|
||||||
|
> exists: the brainstorm replaced it with `resident: all | keys`, and
|
||||||
|
> **`resident: keys` is the ceiling-raising mechanism** — indexes resident, rows
|
||||||
|
> read from the log by offset. It is iteration 2's tasks 5c/5d, not this file's.
|
||||||
|
>
|
||||||
|
> Its premise was also specifically *rejected*, not merely relocated. This
|
||||||
|
> iteration assumed a **user-space resident working set** with faulting and an
|
||||||
|
> eviction policy from [iteration 5](05-bounded-tables-eviction.md). The spec
|
||||||
|
> chose the opposite: no user-space row cache at all, because the kernel page
|
||||||
|
> cache already is one and a `pread` against a cached page is a memcpy — the
|
||||||
|
> position `exploration/postgresql/buffer-and-checkpoint.md` already argued and
|
||||||
|
> the reason the engine avoids `O_DIRECT`.
|
||||||
|
>
|
||||||
|
> **What may still be left:** if measurement after 5c/5d shows the page cache
|
||||||
|
> insufficient for some workload, a user-space working set becomes arguable
|
||||||
|
> again — but only with that number in hand, which is the opposite of how this
|
||||||
|
> file was written. Until then treat the design questions below as answered
|
||||||
|
> elsewhere and the phases as void. Its genuinely durable contribution is its
|
||||||
|
> fork list, especially "does the language surface the fault cost at the *use*
|
||||||
|
> site" — still open, and still the largest question about what writeonce is.
|
||||||
|
>
|
||||||
> **The iteration that actually raises the ceiling, and the one most likely to
|
> **The iteration that actually raises the ceiling, and the one most likely to
|
||||||
> go wrong.** Everything before it makes the limit visible, declared and
|
> go wrong.** Everything before it makes the limit visible, declared and
|
||||||
> managed. This one removes it — for tables that opt in — and in doing so
|
> managed. This one removes it — for tables that opt in — and in doing so
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue