docs: amend principle 7 — the log is authoritative, residency is declared
- driving case: a 120 GB order table on a 32 GB host. Not a tuning problem; no eviction policy fixes it. Developer accepted reconsidering the principle - principle 7 rewritten: durability half UNCHANGED and unconditional (WAL-logged, fsync before ack, CRC-dropped torn tail); residency half demoted from law to per-table declaration. Old wording quoted in place so the amendment is legible, with the reason: a doctrine a real workload cannot satisfy gets ignored, and the failure it produced was an OOM kill - spec: docs/superpowers/specs/2026-08-26-table-residency-design.md One log-structured engine — the WAL already holds every row, so keep an in-RAM id->offset map and pread rows back. No second engine, no user-space row cache (the kernel page cache is the hot copy, which is already this repo's stated position and why it avoids O_DIRECT) - arithmetic that makes it work: 240M rows x 16 B of index = ~3.8 GB resident in 32 GB. Indexes stay resident, rows do not. Buys ~2 orders of magnitude, not infinity — stated plainly in the spec - grammar: two optional keys, `durable: true|false` and `resident: all|index`, both defaulting to today's behaviour, so all 28 existing @table declarations compile untouched and no golden is reblessed - rejected, with reasons recorded: mmap (rows are pointer-bearing — table.c returns (uintptr_t)t as the slot word), buffer pool (the Rust-era phase-12 design that died with that track), paged B-tree (stays rejected), a three-valued enum, automatic spill, disk-backed-by-default - self-review caught the budget defaulting to "none" while promising the ERP developer a diagnostic instead of the OOM killer — contradiction fixed: the budget defaults to a fraction of host memory, and its value comes from databasev2 1's swap-onset measurement - live docs that contradicted the amendment updated (subagent doctrine, its guide, discarded.md's two rows, iteration 04's read claim, 07, 38); dated specs/plans left as records. linkcheck 0 broken / 0 anchors Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
746dc2b42b
commit
da30aa6527
10 changed files with 346 additions and 20 deletions
|
|
@ -14,7 +14,8 @@ You are the database engineer for writeonce's embedded engine.
|
||||||
|
|
||||||
Doctrine (non-negotiable):
|
Doctrine (non-negotiable):
|
||||||
- C11 + libc only. No new dependencies, no atomics on the data path.
|
- C11 + libc only. No new dependencies, no atomics on the data path.
|
||||||
- RAM is authoritative; the WAL makes it durable. An ack means the
|
- The log is authoritative; residency is a declared per-table policy
|
||||||
|
(principle 7, amended 2026-08-26). An ack means the
|
||||||
commit fsynced. Replay is whole-or-not-at-all; torn tails drop.
|
commit fsynced. Replay is whole-or-not-at-all; torn tails drop.
|
||||||
- The engine and the VM heap are two memory worlds crossed only by
|
- The engine and the VM heap are two memory worlds crossed only by
|
||||||
copy (the out-gate: wo_val_decode_vm always copies; rows never hold
|
copy (the out-gate: wo_val_decode_vm always copies; rows never hold
|
||||||
|
|
|
||||||
|
|
@ -68,17 +68,42 @@ binary embeds its own source, so prod is always self-describing.
|
||||||
events; a database that is also the app must not blink.
|
events; a database that is also the app must not blink.
|
||||||
*Enforced by:* [the blue-green spec](superpowers/specs/2026-08-03-blue-green-vm-design.md).
|
*Enforced by:* [the blue-green spec](superpowers/specs/2026-08-03-blue-green-vm-design.md).
|
||||||
|
|
||||||
## 7. RAM is authoritative; the WAL makes it durable
|
## 7. The log is authoritative; residency is a declared per-table policy
|
||||||
|
|
||||||
|
**Amended 2026-08-26.** This principle read "RAM is authoritative; the WAL
|
||||||
|
makes it durable. All reads serve from memory." The durability half was never
|
||||||
|
under strain and is unchanged. The residency half was false for a real
|
||||||
|
workload, so it is now a declaration rather than a law.
|
||||||
|
|
||||||
|
**Durability, unconditional:** every mutation is WAL-logged and fsynced before
|
||||||
|
acknowledgment; boot replays the log; a torn tail is dropped whole by CRC; an
|
||||||
|
ack means the commit reached disk. Mirrors (Postgres) are reconstructible
|
||||||
|
backups that reads and acks never depend on. None of this is per-table and
|
||||||
|
none of it is negotiable.
|
||||||
|
|
||||||
|
**Residency, declared:** what a table keeps in memory is stated at the
|
||||||
|
declaration site. The default keeps every row resident and serves reads at
|
||||||
|
memory speed. A table that cannot fit says so, and then only its indexes are
|
||||||
|
resident while rows are read from the log by offset — the kernel page cache is
|
||||||
|
the hot copy, which is why the engine uses `pread` and deliberately not
|
||||||
|
`O_DIRECT`.
|
||||||
|
|
||||||
|
*Why the amendment:* the original wording is right for a knowledge-management
|
||||||
|
app and simply false for a 120 GB order table on a 32 GB host. A doctrine a
|
||||||
|
real workload cannot satisfy does not get followed, it gets ignored — and the
|
||||||
|
failure it produced was an OOM kill, which is the least debuggable outcome
|
||||||
|
available. The fix keeps one storage engine and one source of truth: the log
|
||||||
|
*is* the database, and RAM is how much of it you choose to serve fast. What was
|
||||||
|
rejected in 2026-08-18 and stays rejected is a *second* engine — a paged
|
||||||
|
B-tree with its own buffer pool ([`plan/discarded.md`](plan/discarded.md)).
|
||||||
|
Reading rows from the log we already write is not that.
|
||||||
|
|
||||||
All reads serve from memory. Every mutation is WAL-logged and fsynced
|
|
||||||
before acknowledgment; boot replays the log. Mirrors (Postgres) are
|
|
||||||
reconstructible backups that reads and acks never depend on.
|
|
||||||
*Why:* one source of truth with predictable latency; durability is a
|
|
||||||
sequential append, not a storage engine bolted to the side.
|
|
||||||
*Enforced by:* [the db-engine binding plan](superpowers/plans/2026-08-01-db-engine-binding.md)
|
*Enforced by:* [the db-engine binding plan](superpowers/plans/2026-08-01-db-engine-binding.md)
|
||||||
(typed WAL + boot replay, shipped); the mirror-is-backup doctrine is
|
(typed WAL + boot replay, shipped); the residency declaration and its
|
||||||
recorded in [`plan/discarded.md`](plan/discarded.md) (the Rust-era WAL
|
enforcement are [databasev2 2](stories/databasev2/02-table-storage-modes.md);
|
||||||
and mirror plans 11/16 were removed with that track 2026-08-18).
|
the mirror-is-backup doctrine is recorded in
|
||||||
|
[`plan/discarded.md`](plan/discarded.md) (the Rust-era WAL and mirror plans
|
||||||
|
11/16 were removed with that track 2026-08-18).
|
||||||
|
|
||||||
## 8. Samples force the grammar
|
## 8. Samples force the grammar
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,8 @@ You are the database engineer for writeonce's embedded engine.
|
||||||
|
|
||||||
Doctrine (non-negotiable):
|
Doctrine (non-negotiable):
|
||||||
- C11 + libc only. No new dependencies, no atomics on the data path.
|
- C11 + libc only. No new dependencies, no atomics on the data path.
|
||||||
- RAM is authoritative; the WAL makes it durable. An ack means the
|
- The log is authoritative; residency is a declared per-table policy
|
||||||
|
(principle 7, amended 2026-08-26). An ack means the
|
||||||
commit fsynced. Replay is whole-or-not-at-all; torn tails drop.
|
commit fsynced. Replay is whole-or-not-at-all; torn tails drop.
|
||||||
- The engine and the VM heap are two memory worlds crossed only by
|
- The engine and the VM heap are two memory worlds crossed only by
|
||||||
copy (the out-gate: wo_val_decode_vm always copies; rows never hold
|
copy (the out-gate: wo_val_decode_vm always copies; rows never hold
|
||||||
|
|
|
||||||
|
|
@ -43,7 +43,7 @@ Status board: [`00-status.md`](../stories/00-status.md) · Doctrine: [`../00-pri
|
||||||
| **Self-hosted `.wo` deploy logic** | Bootstrap problem — the deploy path cannot be written in the language whose deployment it implements. Recorded as a much-later possibility. |
|
| **Self-hosted `.wo` deploy logic** | Bootstrap problem — the deploy path cannot be written in the language whose deployment it implements. Recorded as a much-later possibility. |
|
||||||
| **Script-based / destructive schema migrations (v1)** | Additive-only auto-diff is what makes rollback unconditional: the previous version ignores fields and classes it never knew. Destructive changes reject at propose time. |
|
| **Script-based / destructive schema migrations (v1)** | Additive-only auto-diff is what makes rollback unconditional: the previous version ignores fields and classes it never knew. Destructive changes reject at propose time. |
|
||||||
| **Portability abstractions over Linux** | Targeting one kernel lets the runtime use its sharpest primitives directly instead of the lowest common denominator. Principle 9. |
|
| **Portability abstractions over Linux** | Targeting one kernel lets the runtime use its sharpest primitives directly instead of the lowest common denominator. Principle 9. |
|
||||||
| **Mirrors on the commit path** | The Postgres mirror is a reconstructible backup. RAM is authoritative; reads and acks never depend on it. Principle 7. |
|
| **Mirrors on the commit path** | The Postgres mirror is a reconstructible backup. Reads and acks never depend on it — the log is authoritative. Principle 7. |
|
||||||
|
|
||||||
## Process / docs
|
## Process / docs
|
||||||
|
|
||||||
|
|
@ -72,7 +72,7 @@ goes to find what took each one's place.
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `09-concurrency-scaleout.md` | [`08-shard-actor-runtime.md`](../stories/language-runtime-database/08-shard-actor-runtime.md) + [`11-fibers.md`](../stories/language-runtime-database/11-fibers.md) — the arc, landed 2026-08-21 |
|
| `09-concurrency-scaleout.md` | [`08-shard-actor-runtime.md`](../stories/language-runtime-database/08-shard-actor-runtime.md) + [`11-fibers.md`](../stories/language-runtime-database/11-fibers.md) — the arc, landed 2026-08-21 |
|
||||||
| `10-storage-foundations.md`, `11-wal-and-recovery.md` | [`09-database-engine.md`](../stories/language-runtime-database/09-database-engine.md) (typed WAL + replay) and [`22-durability-throughput-scale.md`](../stories/language-runtime-database/22-durability-throughput-scale.md) (the measurements) |
|
| `10-storage-foundations.md`, `11-wal-and-recovery.md` | [`09-database-engine.md`](../stories/language-runtime-database/09-database-engine.md) (typed WAL + replay) and [`22-durability-throughput-scale.md`](../stories/language-runtime-database/22-durability-throughput-scale.md) (the measurements) |
|
||||||
| `12-engine-disk-cutover.md` | Nothing — RAM stays authoritative by doctrine (principle 7). The disk story is the WAL; reclamation is [`databasev2 3, WAL checkpoint`](../stories/databasev2/03-wal-checkpoint.md) |
|
| `12-engine-disk-cutover.md` | **Partly revisited 2026-08-26.** The disk story is still the WAL and a paged B-tree with its own buffer pool stays rejected. But principle 7's *residency* half was amended: a table may now declare that only its indexes are resident and its rows are read from the log by offset — [databasev2 2](../stories/databasev2/02-table-storage-modes.md). Reclamation remains [`databasev2 3, WAL checkpoint`](../stories/databasev2/03-wal-checkpoint.md) |
|
||||||
| `13-class-model-live-pricing.md` | [`09b-table-relations-query.md`](../stories/language-runtime-database/09b-table-relations-query.md) — `@table`, `ref`/`backlink`, the compiler-checked query surface |
|
| `13-class-model-live-pricing.md` | [`09b-table-relations-query.md`](../stories/language-runtime-database/09b-table-relations-query.md) — `@table`, `ref`/`backlink`, the compiler-checked query surface |
|
||||||
| `07-inotify-content-watcher.md` | [`07-logwatcher-proof.md`](../stories/language-runtime-database/07-logwatcher-proof.md) — the log-watcher sample polls via `fs.stat`; inotify was never surfaced as a builtin |
|
| `07-inotify-content-watcher.md` | [`07-logwatcher-proof.md`](../stories/language-runtime-database/07-logwatcher-proof.md) — the log-watcher sample polls via `fs.stat`; inotify was never surfaced as a builtin |
|
||||||
| `08-sendfile-static-assets.md` | Nothing. `sendfile` is not exposed; static assets are served as `Text` through `net.write` |
|
| `08-sendfile-static-assets.md` | Nothing. `sendfile` is not exposed; static assets are served as `Text` through `net.write` |
|
||||||
|
|
|
||||||
|
|
@ -604,7 +604,7 @@ story file); slots in when scheduled — nothing in the chain depends on it.
|
||||||
|
|
||||||
### ▸ databasev2 — the database beyond RAM
|
### ▸ databasev2 — the database beyond RAM
|
||||||
|
|
||||||
New 2026-08-26. **The problem:** RAM is authoritative (principle 7) and nothing
|
New 2026-08-26. **The problem:** rows were resident unconditionally and nothing
|
||||||
declares a budget. Rows live in `malloc`'d slabs whose addresses are stable
|
declares a budget. Rows live in `malloc`'d slabs whose addresses are stable
|
||||||
forever; there is no eviction, spill or paging anywhere in `database/src/`; the
|
forever; there is no eviction, spill or paging anywhere in `database/src/`; the
|
||||||
WAL never checkpoints so boot replays all history; and durability is one
|
WAL never checkpoints so boot replays all history; and durability is one
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,14 @@ status: refine
|
||||||
> global switch forces "everything is precious" or "nothing is", and the
|
> global switch forces "everything is precious" or "nothing is", and the
|
||||||
> developer pays for the wrong one either way.
|
> developer pays for the wrong one either way.
|
||||||
|
|
||||||
|
> **⚠ Superseded in part, 2026-08-26.** The brainstorm settled on a different
|
||||||
|
> shape than this document describes: one log-structured engine where the WAL
|
||||||
|
> *is* the row store, with residency declared per table (`resident: all` /
|
||||||
|
> `resident: index`) rather than a three-valued `mode:` enum including `cold`.
|
||||||
|
> Principle 7 was amended accordingly — the log is authoritative, residency is
|
||||||
|
> the declaration. This file is rewritten once the grammar is approved; read the
|
||||||
|
> [track index](00-story.md) and `docs/00-principles.md` §7 as current.
|
||||||
|
|
||||||
## Goals
|
## Goals
|
||||||
|
|
||||||
- **Move the storage decision to the declaration site.** `@table(mode: ...)`,
|
- **Move the storage decision to the declaration site.** `@table(mode: ...)`,
|
||||||
|
|
|
||||||
|
|
@ -93,9 +93,14 @@ chain: 5
|
||||||
- **io_uring for the network/accept path.** This iteration is the WAL write
|
- **io_uring for the network/accept path.** This iteration is the WAL write
|
||||||
path only; the socket side is the shard-actor runtime's and the network
|
path only; the socket side is the shard-actor runtime's and the network
|
||||||
layer's concern.
|
layer's concern.
|
||||||
- **io_uring for reads.** RAM is authoritative — reads never touch a
|
- **io_uring for reads.** For a fully-resident table reads never touch a
|
||||||
descriptor (phase-B doctrine), so there is nothing to accelerate on the
|
descriptor, so there is nothing to accelerate on the read path. This is a
|
||||||
read path. This is a write-durability optimization, full stop.
|
write-durability optimization, full stop. **Note (2026-08-26):** principle 7's
|
||||||
|
residency half was amended, so a table declaring `resident: index`
|
||||||
|
([databasev2 2](02-table-storage-modes.md)) *does* `pread` rows from the log —
|
||||||
|
and accelerating that read path with io_uring becomes a real, separate
|
||||||
|
question. It is not this iteration's, and it should not be folded in: this
|
||||||
|
one is about the commit path and its acceptance is a durability number.
|
||||||
- **Registered buffers / fixed files / SQPOLL tuning** beyond what the
|
- **Registered buffers / fixed files / SQPOLL tuning** beyond what the
|
||||||
benchmark shows is worth it. Start with the plain submit/complete model;
|
benchmark shows is worth it. Start with the plain submit/complete model;
|
||||||
add ring features only when 22's number says a specific one pays.
|
add ring features only when 22's number says a specific one pays.
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,7 @@ status: refine
|
||||||
> **Inserted 2026-08-22** (developer ask: "can the persistent db be in
|
> **Inserted 2026-08-22** (developer ask: "can the persistent db be in
|
||||||
> file.db form?"). The truth is already almost there: `WO_DATA=<dir>`
|
> file.db form?"). The truth is already almost there: `WO_DATA=<dir>`
|
||||||
> holds exactly ONE file (`shard-0.wal`) — the entire persistent state,
|
> holds exactly ONE file (`shard-0.wal`) — the entire persistent state,
|
||||||
> since RAM is authoritative and no data pages exist. This iteration
|
> since the log is the whole store and no data pages exist. This iteration
|
||||||
> makes the surface say so: point `WO_DATA` at a file and THAT file is
|
> makes the surface say so: point `WO_DATA` at a file and THAT file is
|
||||||
> the store. Small, driver-only, independent of the concurrency chain.
|
> the store. Small, driver-only, independent of the concurrency chain.
|
||||||
|
|
||||||
|
|
@ -50,7 +50,7 @@ status: refine
|
||||||
|
|
||||||
## Out Of Scope
|
## Out Of Scope
|
||||||
|
|
||||||
- A paged database file (SQLite's shape) — RAM is authoritative; the
|
- A paged database file (SQLite's shape) — still rejected; the
|
||||||
disk story is the WAL, full stop.
|
disk story is the WAL, full stop.
|
||||||
- Checkpoint/compaction — [iteration 32](03-wal-checkpoint.md)'s; its
|
- Checkpoint/compaction — [iteration 32](03-wal-checkpoint.md)'s; its
|
||||||
rename-swap (write snapshot+tail to a NEW file, fsync, `rename()`
|
rename-swap (write snapshot+tail to a NEW file, fsync, `rename()`
|
||||||
|
|
|
||||||
|
|
@ -116,7 +116,7 @@ status: refine
|
||||||
- **A plugin/app ecosystem.** In-runtime recompile is
|
- **A plugin/app ecosystem.** In-runtime recompile is
|
||||||
[iteration 26](26-blue-green-deploy.md)'s; nothing here loads
|
[iteration 26](26-blue-green-deploy.md)'s; nothing here loads
|
||||||
code at run time.
|
code at run time.
|
||||||
- **Scale claims.** RAM is authoritative (64 MiB arena by default,
|
- **Scale claims.** Rows are resident by default (64 MiB arena default,
|
||||||
`WO_HEAP_MB`), so this is a team-scale platform and the README states the
|
`WO_HEAP_MB`), so this is a team-scale platform and the README states the
|
||||||
row ceiling it was measured at.
|
row ceiling it was measured at.
|
||||||
|
|
||||||
|
|
|
||||||
286
docs/superpowers/specs/2026-08-26-table-residency-design.md
Normal file
286
docs/superpowers/specs/2026-08-26-table-residency-design.md
Normal file
|
|
@ -0,0 +1,286 @@
|
||||||
|
# databasev2 2 — table residency: the log as row store (for review)
|
||||||
|
|
||||||
|
> **Status:** design approved in brainstorm 2026-08-26, pending review of this
|
||||||
|
> document. Implements [databasev2 2](../../stories/databasev2/02-table-storage-modes.md),
|
||||||
|
> which is rewritten to match once this is approved. Amends
|
||||||
|
> [principle 7](../../00-principles.md) — already applied.
|
||||||
|
>
|
||||||
|
> Driving requirement, from the developer: **a 120 GB order table on a 32 GB
|
||||||
|
> host.** Not a tuning problem; no eviction policy fixes it. Small data-driven
|
||||||
|
> applications are well served by today's resident default and must not regress.
|
||||||
|
|
||||||
|
## Decisions taken (the brainstorm's forks, settled)
|
||||||
|
|
||||||
|
| Fork | Decision |
|
||||||
|
| --- | --- |
|
||||||
|
| One enum or two keys | **Two keys.** `durable:` and `resident:` answer two different developer questions ("do I need this after a restart?", "does it fit in RAM?"). One enum forces a name for each *combination*, which is what made a third value unreadable. |
|
||||||
|
| Mode vocabulary | **`resident: all \| index`** and **`durable: true \| false`**. No `cold`, `tiered`, `paged`, `mmap` or `buffer` in the grammar. |
|
||||||
|
| Optional or mandatory | **Optional, both default to today's behaviour** (`durable: true`, `resident: all`). All 28 existing declarations compile unchanged; no goldens reblessed. |
|
||||||
|
| Which storage architecture | **One engine, log-structured.** The WAL already holds every row; keep an in-RAM id→offset map and read rows back with `pread`. No second engine. |
|
||||||
|
| Row cache | **None in user space.** The kernel page cache is the hot copy — the repo's own stated position in `exploration/postgresql/buffer-and-checkpoint.md`: "`pread` against an fd that already has its page cached is a memcpy… the page cache is the one cache we want", and the reason the engine avoids `O_DIRECT`. |
|
||||||
|
| `@unique` on a non-resident table | **Allowed; its index is unconditionally resident.** Settled here rather than deferred — see Constraints. |
|
||||||
|
| Budget unit | **Bytes** (estimated resident footprint). Rows is the meaningless unit: a text-heavy row and an Int-only row differ by an order of magnitude, so a row count cannot bound RAM. |
|
||||||
|
| Rejected architectures | `mmap` and a buffer pool stay out — see Alternatives rejected. `discarded.md`'s paged-engine rejection is amended to *partly revisited*, not reversed. |
|
||||||
|
|
||||||
|
## The problem, read off the engine
|
||||||
|
|
||||||
|
Facts, each verified in source rather than assumed:
|
||||||
|
|
||||||
|
- Rows live in `malloc`'d slabs of `DB_SLAB_ROWS` (256), allocated as a table
|
||||||
|
grows and freed only at table teardown. The free-slot list recycles removed
|
||||||
|
slots, so a delete-heavy table plateaus; a growing table only grows.
|
||||||
|
- **The ceiling is process RSS and nothing declares it.** `WO_HEAP_MB`
|
||||||
|
(default 64 MiB) bounds the VM object arena; table storage is separate
|
||||||
|
`malloc`. No knob says "this database may use at most N".
|
||||||
|
- No eviction, spill, paging or LRU exists anywhere in `database/src`.
|
||||||
|
- **Durability is process-global.** `main.c` opens one `shard-0.wal` when
|
||||||
|
`WO_DATA` is set. `db.c` guards every WAL append with a null check on
|
||||||
|
`vm->rt.wal`, so with no `WO_DATA` **every table is silently volatile** — a
|
||||||
|
program can declare nothing and lose everything.
|
||||||
|
- An allocation failure is clean: every `malloc` in the row encoder is checked
|
||||||
|
and `DB_ERR_OOM` maps to `WO_T_OOM`, a catchable trap. The dangerous exit is
|
||||||
|
the one *before* that — swap thrash, which carries no error signal at all.
|
||||||
|
|
||||||
|
The measurements that bound the design, from iteration 22: durable inserts
|
||||||
|
≈4.5k/s against RAM ≈297k/s (the 66× fsync gap); reads 1.3M ops/s at p50 1µs
|
||||||
|
after the index probe landed. The read number is what a resident table buys and
|
||||||
|
what must not regress for tables that keep it.
|
||||||
|
|
||||||
|
### Why the arithmetic works
|
||||||
|
|
||||||
|
A 120 GB table at ~500 B/row is ≈240M rows. An id→offset entry is 16 bytes, so
|
||||||
|
the resident index is **≈3.8 GB** — comfortable inside 32 GB with room for two
|
||||||
|
secondary indexes of similar cost. That ratio is the whole design: **indexes
|
||||||
|
stay resident, rows do not.** It buys roughly two orders of magnitude of table
|
||||||
|
size, not infinity, and the spec says so plainly because a design sold as
|
||||||
|
unlimited gets deployed as if it were.
|
||||||
|
|
||||||
|
## The design
|
||||||
|
|
||||||
|
### Grammar
|
||||||
|
|
||||||
|
Two new `@table` arguments, both optional. The parser's argument loop already
|
||||||
|
matches `name` and `index` and rejects anything else with a catalogued
|
||||||
|
diagnostic; these are two more arms and an updated message. `Ast.table_cfg`
|
||||||
|
grows two fields beside `table_name` and `indexes`.
|
||||||
|
|
||||||
|
| Argument | Values | Default | Meaning |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `durable` | `true`, `false` | `true` | `false` skips the WAL append entirely: no record, no fsync, ack from RAM, table empty after restart. |
|
||||||
|
| `resident` | `all`, `index` | `all` | `index` keeps the id map and every secondary index in RAM; rows are read from the log by offset. |
|
||||||
|
|
||||||
|
`true`/`false` are already keyword tokens; `all`/`index` are parsed as the same
|
||||||
|
bare identifiers the `index:` argument's column list already accepts. No lexer
|
||||||
|
change.
|
||||||
|
|
||||||
|
**One wart, surfaced rather than buried:** `resident: index` puts the word
|
||||||
|
`index` in value position while `index:` is also a key, so
|
||||||
|
`@table(index: [customer], resident: index)` reads awkwardly on first
|
||||||
|
encounter. The parser distinguishes them structurally and there is no
|
||||||
|
ambiguity, but a reviewer may prefer `resident: keys` or `resident: index_only`.
|
||||||
|
Flagged for the review of this document; the mechanism is unaffected either
|
||||||
|
way.
|
||||||
|
|
||||||
|
### The four combinations
|
||||||
|
|
||||||
|
| `durable` | `resident` | Status |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `true` | `all` | Today's behaviour. The default. Reads at memory speed. |
|
||||||
|
| `false` | `all` | Volatile scratch: sessions, rate-limit counters, idempotency keys. Skips the 66× fsync cost. What porch 1–3 need. |
|
||||||
|
| `true` | `index` | The 120 GB case. Rows in the log, indexes resident, `pread` on read. |
|
||||||
|
| `false` | `index` | **Refused at compile time.** Rows would have nowhere to be read from. |
|
||||||
|
|
||||||
|
### Read, write and recovery paths
|
||||||
|
|
||||||
|
- **Insert** — unchanged for `resident: all`. For `resident: index`: encode and
|
||||||
|
append the record as today, then record id→offset in the resident map instead
|
||||||
|
of retaining the row in a slab. The WAL append is already the durable write;
|
||||||
|
this stops discarding its payload.
|
||||||
|
- **Read by id** — resident map lookup, then `pread` at the offset, verify CRC,
|
||||||
|
decode into fresh VM values. The decode path already exists
|
||||||
|
(`wo_val_decode_vm` always copies; rows never hand out interior pointers), so
|
||||||
|
the change is where the bytes come from.
|
||||||
|
- **Update** — append a new record, repoint the offset. The superseded record
|
||||||
|
becomes garbage, reclaimed by the checkpoint.
|
||||||
|
- **Delete** — append a tombstone, drop the id from the map and every index.
|
||||||
|
- **Scan** — a sequential walk of the log, which is the case log-structured
|
||||||
|
storage is best at. Cost changes from memory-speed to sequential-disk; the
|
||||||
|
query surface is unchanged in *meaning* and materially different in *cost*.
|
||||||
|
- **Boot** — replay rebuilds the offset map by scanning the log. Correct, and
|
||||||
|
O(entire history), which is the honest cost of shipping this before
|
||||||
|
[databasev2 3](../../stories/databasev2/03-wal-checkpoint.md).
|
||||||
|
|
||||||
|
### Row encoding: the one real rewrite
|
||||||
|
|
||||||
|
A row slot today holds raw pointers. `table.c`'s `WO_K_TEXT` case allocates a
|
||||||
|
`db_text` and returns its address as the slot word; owned, multi and map do the
|
||||||
|
same. Pointers minted by a dead process are meaningless in a file, so the
|
||||||
|
on-disk record must be **self-contained and offset-based**: every heap value
|
||||||
|
inlined into the record with internal references expressed as offsets from the
|
||||||
|
record's own start.
|
||||||
|
|
||||||
|
This is confined to `db_val_encode`/`db_val_decode` and the record framing. It
|
||||||
|
is the substantive engineering in this iteration and the place to expect the
|
||||||
|
bugs. `wal.c` already frames records as `len|crc|payload|mark`, so the framing
|
||||||
|
exists; what changes is that the payload must be readable standalone rather than
|
||||||
|
only replayable.
|
||||||
|
|
||||||
|
### Constraints across the residency boundary
|
||||||
|
|
||||||
|
- **`@unique` is allowed, and its index is unconditionally resident.** A shadow
|
||||||
|
check cannot scan slabs that are not there, so correctness requires the unique
|
||||||
|
column's index in RAM. Cost is one hash entry per row — the same order as the
|
||||||
|
id map, so a unique column roughly doubles the resident index. Stated at the
|
||||||
|
declaration so the developer can see what they bought. Silently checking only
|
||||||
|
resident rows is the one outcome that must never ship.
|
||||||
|
- **Foreign-key restrict works unchanged.** It is a secondary-index probe, and
|
||||||
|
secondary indexes are resident.
|
||||||
|
- **`ref` navigation works unchanged**, at the cost of a `pread` per hop.
|
||||||
|
- **A `resident: all` table may hold a `ref` into a `resident: index` table**
|
||||||
|
and vice versa — both are durable, so neither evaporates. This is the case
|
||||||
|
that a `durable: false` table genuinely breaks, below.
|
||||||
|
|
||||||
|
### What the compiler enforces
|
||||||
|
|
||||||
|
Four refusals, each a catalogued `WO-E1xx` diagnostic added in the same change
|
||||||
|
as the code — not afterwards:
|
||||||
|
|
||||||
|
1. `durable: false` with `resident: index` — the meaningless combination.
|
||||||
|
2. An unknown value for either argument, or either argument given twice.
|
||||||
|
3. **A `durable: true` table holding a `ref` into a `durable: false` table.**
|
||||||
|
A persistent row cannot reference one that evaporates on restart; FK restrict
|
||||||
|
cannot save it and the dangling reference is provable from the class table.
|
||||||
|
This is the highest-value check in the iteration.
|
||||||
|
4. A `cold`, `tiered`, `ram` or other retired mode word — refused with a message
|
||||||
|
naming the two real arguments, so the vocabulary explored during the
|
||||||
|
brainstorm does not become folklore.
|
||||||
|
|
||||||
|
### What the runtime enforces
|
||||||
|
|
||||||
|
- **`durable: true` with no `WO_DATA` is a startup refusal.** Today this
|
||||||
|
combination silently loses everything, which is the worst failure mode in the
|
||||||
|
current engine. A program that declares durability and is given nowhere to put
|
||||||
|
it must not start. This is independent of residency and is arguably the most
|
||||||
|
valuable single line in the spec.
|
||||||
|
- **A byte budget that exists by default, breached loudly.** The budget bounds
|
||||||
|
estimated resident footprint across all tables. On breach the program refuses
|
||||||
|
with a message naming the largest offending table and the exact annotation to
|
||||||
|
add, so the 120 GB developer meets a diagnostic at 32 GB rather than the OOM
|
||||||
|
killer.
|
||||||
|
|
||||||
|
**The default must not be "no budget"** — that was a contradiction in the
|
||||||
|
first draft of this spec: a budget nobody sets cannot produce the diagnostic
|
||||||
|
that is this design's main deliverable, and the ERP developer would still meet
|
||||||
|
the OOM killer. So the default is a **fraction of host-detected available
|
||||||
|
memory**, overridable by an environment variable and by a per-program
|
||||||
|
declaration. Choosing that fraction is the one number this spec cannot supply:
|
||||||
|
it comes from [databasev2 1](../../stories/databasev2/01-ram-ceiling-measurement.md)'s
|
||||||
|
swap-onset measurement, which is why 1 sequences before 2. Until 1 lands,
|
||||||
|
implement the mechanism with a conservative placeholder fraction and treat the
|
||||||
|
value as unset rather than settled.
|
||||||
|
|
||||||
|
Consequence to accept deliberately: a program that today grows past that
|
||||||
|
fraction and survives on a large host will now refuse. That is the intended
|
||||||
|
behaviour change — it converts an invisible slide into swap into a startup
|
||||||
|
error — but it *is* a behaviour change, and it is the second of the two
|
||||||
|
breaks listed under Migration.
|
||||||
|
- `WO_DATA` remains the data-directory location. It stops being the durability
|
||||||
|
switch; the declaration is.
|
||||||
|
|
||||||
|
### Format
|
||||||
|
|
||||||
|
The class descriptor carries both properties, so the runtime never re-derives
|
||||||
|
them. This moves `WOB_VERSION` (currently 6) and the format contract in the same
|
||||||
|
commit as the code. An older image is refused on version rather than misread.
|
||||||
|
|
||||||
|
## Why this shape
|
||||||
|
|
||||||
|
- **It keeps one engine and one source of truth.** The log *is* the database.
|
||||||
|
Nothing here adds a second storage system, which is what the 2026-08-18
|
||||||
|
rejection was actually about.
|
||||||
|
- **It costs nothing for tables that do not use it.** A `resident: all` table
|
||||||
|
takes the same path it takes today. The 1.3M ops/s read baseline is the
|
||||||
|
regression gate, and a measurable regression there is grounds to reject the
|
||||||
|
implementation rather than tune it.
|
||||||
|
- **The failure mode becomes a diagnostic.** Two of the three exits
|
||||||
|
characterised in databasev2 1 — swap thrash and the OOM killer — are replaced
|
||||||
|
by a refusal that names the fix.
|
||||||
|
- **It is declared, not automatic.** No threshold heuristic, no performance
|
||||||
|
cliff the compiler cannot explain. Consistent with a language whose thesis is
|
||||||
|
that the compiler tells you the truth.
|
||||||
|
|
||||||
|
## Alternatives rejected
|
||||||
|
|
||||||
|
| Alternative | Why not |
|
||||||
|
| --- | --- |
|
||||||
|
| **`mmap` the row file** | Requires offset-based rows *and* gives up precise ack-after-fsync for the kernel's flush schedule. The repo's own mmap study only ever proposed it read-only for segment lookups. If rows become self-contained anyway, mmap is a possible later optimisation of the read path — recorded, not adopted. |
|
||||||
|
| **Buffer pool with dirty-page tracking** | This is the Rust-era phase-12 design in `exploration/postgresql/buffer-and-checkpoint.md` (`CachedRow { bytes, dirty }`, `WO_CACHE_ROWS` LRU) that died with that track. It duplicates the kernel page cache, and the page cache is explicitly the cache this project wants. |
|
||||||
|
| **Paged B-tree engine** | Rejected 2026-08-18 and still rejected. Reading rows from the log we already write is not this. |
|
||||||
|
| **A three-valued `mode:` enum** | Needs a name per combination. The brainstorm demonstrated that the third name is unwriteable before its mechanism is decided. |
|
||||||
|
| **Automatic spill at a threshold** | No annotation needed, but unpredictable cliffs and magic the compiler cannot explain. |
|
||||||
|
| **Disk-backed by default** | Costs every application the 1µs read, including the small ones explicitly said to be well served today. |
|
||||||
|
|
||||||
|
## Proof plan
|
||||||
|
|
||||||
|
Acceptance is the story's Given/When/Then list; this is how each is exercised.
|
||||||
|
|
||||||
|
- **Compatibility** — all 28 existing `@table` declarations compile untouched;
|
||||||
|
`just employee`, `just db-actor`, `just web-app`, `just site`, `oop-accept`
|
||||||
|
unchanged; no golden reblessed.
|
||||||
|
- **Volatility** — a `durable: false` table produces no WAL growth (measured,
|
||||||
|
not asserted) and is empty after restart while durable siblings replay intact.
|
||||||
|
- **Residency correctness** — a `resident: index` table larger than the
|
||||||
|
configured budget returns every row correctly by id, byte-identical including
|
||||||
|
every heap-valued column, and scans in full.
|
||||||
|
- **Constraints across the boundary** — `@unique` refuses a duplicate whose
|
||||||
|
conflicting row is not resident; FK restrict refuses a delete whose only
|
||||||
|
referrer is not resident. Corpus fixtures, both.
|
||||||
|
- **Compiler refusals** — a `compile-fail` fixture per diagnostic, each pinning
|
||||||
|
the exact code.
|
||||||
|
- **Runtime refusals** — `durable: true` with no `WO_DATA` fails at startup;
|
||||||
|
a budget breach names the table and the annotation.
|
||||||
|
- **Crash safety** — `kill -9` mid-append and mid-checkpoint on a
|
||||||
|
`resident: index` table; replay loses no acked write and no row appears twice.
|
||||||
|
- **Performance** — new baseline rows for the `resident: index` read path with
|
||||||
|
its amplification versus resident, published in `perf-targets.md` as a number
|
||||||
|
a developer can plan around; and a regression check that resident tables did
|
||||||
|
not move.
|
||||||
|
- **Sanitisers** — ASan on the new decode path, which is where the bugs are.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- **Checkpoint and compaction** — [databasev2 3](../../stories/databasev2/03-wal-checkpoint.md).
|
||||||
|
This spec's boot cost is O(history) until 3 lands, and 3's snapshot should
|
||||||
|
persist the offset map so boot stops rescanning. That coupling is stated in
|
||||||
|
both documents.
|
||||||
|
- **Eviction policy and a resident row cache** — [databasev2 5](../../stories/databasev2/05-bounded-tables-eviction.md).
|
||||||
|
This design needs neither: rows are either all resident or none are.
|
||||||
|
- **io_uring on the read path** — a real question that only exists after this
|
||||||
|
lands; noted in [databasev2 4](../../stories/databasev2/04-io-uring-commit.md),
|
||||||
|
not folded into it.
|
||||||
|
- **Per-shard residency** — the owner shard owns the store and the log. A
|
||||||
|
`durable: false` table arguably need not live on the owner at all, which would
|
||||||
|
be dramatically faster and a different consistency story. Recorded as a
|
||||||
|
candidate, deliberately not decided here.
|
||||||
|
- **Migrating an existing dataset between settings**, and a WAL written when a
|
||||||
|
table had different settings: refuse clearly on mismatch, do not convert.
|
||||||
|
- **`transaction { }` and `@table` feature flags** — language iteration 18,
|
||||||
|
approved spec, left whole.
|
||||||
|
|
||||||
|
## Migration
|
||||||
|
|
||||||
|
Nothing to convert. Both arguments default to present behaviour, so every
|
||||||
|
existing program, manifest, corpus fixture and golden compiles and runs
|
||||||
|
unchanged.
|
||||||
|
|
||||||
|
**Two deliberate behaviour changes**, both converting a silent failure into a
|
||||||
|
loud one, and both listed here so neither arrives as a surprise:
|
||||||
|
|
||||||
|
1. `durable: true` (the default) with no `WO_DATA` becomes a startup refusal.
|
||||||
|
Today it silently discards every write.
|
||||||
|
2. Total estimated resident footprint crossing the default budget fraction
|
||||||
|
becomes a startup or insert refusal. Today the program slides into swap with
|
||||||
|
no signal and is eventually killed.
|
||||||
|
|
||||||
|
Both are opt-out-able by explicit declaration. Neither is a data-format change,
|
||||||
|
so a rollback is a binary swap with no migration.
|
||||||
Loading…
Reference in a new issue