shoney.arickathil
746dc2b42b
- docs/stories/databasev2/, numbered from 1. Six PENDING database iterations
moved from the language track and renumbered, keeping the old id in
`was_language_iteration:` so a search for "iteration 32" still finds it:
32 -> 3 WAL checkpoint, 23 -> 4 io_uring commit, 33 -> 7 single-file store,
27 -> 8 query grammar, 20 -> 9 cross-program, 21 -> 10 keypair auth.
Done work (9, 9b, 22) stays as v1 history; language 18 left whole
- the problem, read off the engine not guessed: rows are malloc'd slabs with
addresses stable forever, NO eviction/spill/paging anywhere in database/src,
the WAL never checkpoints so boot replays all history, and durability is one
process-global WO_DATA so no table can say it matters more than another.
An allocation failure IS a clean catchable WO_T_OOM — but swap thrash
arrives first and carries no error signal at all, which is the real hazard
- four new iterations:
1 measure the ceiling FIRST (curve not cliff; the three exits; kill -9 at
exhaustion) — every later default should follow from a number
2 `@table(mode: ram | durable | cold)` — the grammar ask. Small surface
(Ast.table_cfg gains a key, the parser already rejects unknown args), big
semantics: `durable` defaults so nothing changes silently, and the
compiler refuses a durable row holding a `ref` into a ram table
5 bounded tables + refuse/evict/back-pressure, shedding BEFORE the OS acts
6 cold tiering — mostly forks, incl. whether the language surfaces the
fault cost and whether @unique on cold is refused outright. A paged
B-tree stays rejected: if tiering needs one, reject tiering
- 39 links repointed, link TEXT renumbered to track-local ids; arc gains one
pointer row replacing the six moved; board + board-views cover three tracks
- linkcheck 0 broken / 0 anchors; no code blocks in any story
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
188 lines
11 KiB
Markdown
188 lines
11 KiB
Markdown
---
|
||
track: databasev2
|
||
iteration: "2"
|
||
status: refine
|
||
---
|
||
|
||
# databasev2 2 — `@table` storage modes: durability becomes a language decision
|
||
|
||
> Part of [Story — databasev2: the database beyond RAM](00-story.md).
|
||
> Needs [1](01-ram-ceiling-measurement.md) — a mode's default should follow from
|
||
> a measurement, not a preference.
|
||
>
|
||
> **The developer's ask, and the language enrichment this track exists for.**
|
||
> Today durability is one environment variable for a whole process: `WO_DATA` is
|
||
> set and every `@table` is WAL-logged, or it is not and none are
|
||
> (`runtime/src/main.c`). Real applications are not uniform. A session table, a
|
||
> rate-limit counter and a page cache are resident and disposable; an orders
|
||
> table is resident and precious; an audit log is precious and rarely read. One
|
||
> global switch forces "everything is precious" or "nothing is", and the
|
||
> developer pays for the wrong one either way.
|
||
|
||
## Goals
|
||
|
||
- **Move the storage decision to the declaration site.** `@table(mode: ...)`,
|
||
chosen per table, in the source, where the person who knows what the data is
|
||
worth is already writing. An operations runbook is the wrong place for a fact
|
||
the compiler could hold.
|
||
- **Three modes, each earning its existence.** `ram` — resident, never logged,
|
||
gone on restart. `durable` — today's behaviour, resident and ack-after-fsync,
|
||
and the **default** so every existing program is byte-identical. `cold` —
|
||
durable and not required to be resident, declared here and *implemented* in
|
||
[6](06-cold-tiering.md), because a mode with no engine behind it is a promise.
|
||
- **Make the compiler enforce what the mode means.** This is the part that makes
|
||
it a language feature rather than a config key. A program that inserts into a
|
||
`ram` table and expects the row after a restart is stating a contradiction, and
|
||
the compiler is the right place to say so — at minimum for the cases it can
|
||
see statically, with the diagnostic catalogued like every other.
|
||
- **Let the engine act on it.** `ram` tables skip the WAL write entirely, which
|
||
is not merely a saving — it is the 66× gap iteration 22 measured (4.5k durable
|
||
vs 297k RAM inserts/s) becoming available per table instead of per process.
|
||
They also become the first candidates to shed under pressure, which is what
|
||
[5](05-bounded-tables-eviction.md) builds on.
|
||
- **Keep principle 7 intact and say why.** RAM stays authoritative for every
|
||
table that says so. `cold` is a declared, per-table exception a developer opts
|
||
into with the trade visible at the declaration.
|
||
|
||
## Phases
|
||
|
||
### Phase A — the grammar
|
||
|
||
- Extend the `@table` argument parser with `mode:`. The path is already cut:
|
||
the argument loop matches `name` and `index` and rejects anything else with a
|
||
catalogued diagnostic (`unknown @table argument ... (supported: name, index)`),
|
||
so this is one more arm plus an updated message.
|
||
- Extend `Ast.table_cfg` — today `{ table_name; indexes }` — with the mode, and
|
||
give it the default so every existing `@table` keeps its meaning.
|
||
- Reject the incoherent cases at parse time: `mode` given twice, an unknown mode
|
||
name. Both belong in the same diagnostic family as the existing `@table`
|
||
errors, and both go in the error catalog **in this change**, not later — that
|
||
catalog went stale once by exactly that omission.
|
||
- Verify: golden AST fixtures for each mode; `woc-test` green; every existing
|
||
`@table` in every sample parses unchanged.
|
||
|
||
### Phase B — the mode reaches the image and the engine
|
||
|
||
- Carry the mode through the class descriptor into the `.wob` image so the
|
||
runtime knows it without re-deriving anything. This is a format change, so it
|
||
moves `WOB_VERSION` and the format contract in the same commit as the code.
|
||
- The engine consults it at the one choke point that already exists: the
|
||
`INDEX HOOK` / WAL staging site in `wo_row_insert` / `wo_row_remove`, which
|
||
`database/src/CODE-LOGIC.md` names as the only place storage may be mutated.
|
||
A `ram` table stages nothing.
|
||
- Replay must skip records for tables that are now `ram` — a WAL written when a
|
||
table was `durable` and replayed after the source changed is a real
|
||
migration case, and silently resurrecting rows into a `ram` table would be
|
||
worse than refusing.
|
||
- Verify: a `ram` table's inserts produce no WAL growth (measured, not assumed);
|
||
a `durable` table is byte-identical to today; a mode change across a restart
|
||
is handled explicitly rather than by accident.
|
||
|
||
### Phase C — the compiler's enforcement
|
||
|
||
- Decide how far static checking goes (fork 2) and implement that much. The
|
||
floor: `WO_DATA` set with every table `ram` is a program that asked for a data
|
||
directory it will never write to — worth a warning at least.
|
||
- The interesting case is a `ram` table participating in a `ref`/`backlink`
|
||
relation with a `durable` one. A durable row holding a foreign key into a
|
||
table that evaporates on restart is a dangling reference by construction, and
|
||
FK-restrict cannot save it. This is the check most worth having, and it is
|
||
statically visible from the class table.
|
||
- Verify: corpus `compile-fail` fixtures for each refusal; each carries the
|
||
exact `WO-E###` the catalog now documents.
|
||
|
||
### Phase D — prove it on a real workload
|
||
|
||
- Give `docs/examples/employee` or the `db-bench` sample a mixed schema — at
|
||
least one `ram` table and one `durable` — and gate the distinction: after a
|
||
restart, the durable rows are present and the ram rows are gone. That single
|
||
assertion is the whole feature.
|
||
- Extend the durability legs of `just db-bench` so the per-table write-path
|
||
saving appears as a baseline number, not a claim.
|
||
- Verify: `just employee`, `just db-actor`, `just db-bench`, `oop-accept` green.
|
||
|
||
### Phase E — document the contract
|
||
|
||
- The `@table` mode surface in the language-surface guide and the db-binding
|
||
contract; the new diagnostics in the error catalog; the mode's effect on
|
||
replay in `database/src/CODE-LOGIC.md`.
|
||
- Verify: `just linkcheck` clean; the language-surface guide's `@table` row
|
||
matches what the parser actually accepts.
|
||
|
||
## Acceptance Criteria
|
||
|
||
- **Given** every existing `@table` declaration in the repository, **when** it is
|
||
compiled after this change, **then** behaviour is byte-identical — `durable`
|
||
is the default and nothing opts in silently.
|
||
- **Given** a table declared `mode: ram`, **when** rows are inserted with
|
||
`WO_DATA` set, **then** the WAL does not grow, and after a restart the table is
|
||
empty while `durable` tables in the same program replay intact.
|
||
- **Given** `mode: ram` and a measured insert workload, **when** it runs against
|
||
the same shape as a `durable` table, **then** the write-path saving is visible
|
||
in `bench/baseline.json` — the per-table half of iteration 22's 66× gap.
|
||
- **Given** an unknown mode name or `mode:` given twice, **when** it is compiled,
|
||
**then** it fails with the catalogued diagnostic naming the legal modes.
|
||
- **Given** a `durable` table holding a `ref` into a `ram` table, **when** it is
|
||
compiled, **then** the compiler refuses (or warns, per fork 2) — a persistent
|
||
row cannot reference one that evaporates.
|
||
- **Given** a WAL containing records for a table whose source now says `ram`,
|
||
**when** the program starts, **then** the situation is handled explicitly
|
||
(refuse, or skip and report) and never by silently loading rows into a table
|
||
declared not to have any.
|
||
- **Given** the `.wob` format change, **when** an image from the previous version
|
||
is loaded, **then** the loader refuses it clearly on the version rather than
|
||
misreading a descriptor.
|
||
|
||
## Out Of Scope
|
||
|
||
- **Implementing `cold`.** Declared here so the mode set is settled and the
|
||
format carries it; the engine behaviour is [6](06-cold-tiering.md). Until then
|
||
a `cold` declaration must be refused rather than silently treated as
|
||
`durable` — accepting a mode that does nothing is how a feature becomes a lie.
|
||
- **Per-table capacity limits and eviction** — [5](05-bounded-tables-eviction.md).
|
||
This iteration says what a table *is*; that one says how much of it there may
|
||
be.
|
||
- **`@table` feature flags and `transaction { }`** — language
|
||
[iteration 18](../language-runtime-database/18-memory-db-features.md), whose
|
||
spec is approved and deliberately left whole.
|
||
- **Per-table WAL files.** One log, one writer, shard 0 — the invariant stage 3
|
||
established and [7](07-single-file-db.md) depends on. Modes decide *whether* a
|
||
table logs, never *where*.
|
||
- **Migrating an existing dataset between modes.** A schema-change story, and
|
||
the repo already records destructive migrations as a recorded future.
|
||
- **Encryption at rest, compression of the WAL.** Neither has a consumer.
|
||
|
||
## Info
|
||
|
||
The grammar surface this touches, read from the source: the parser's `@table`
|
||
argument loop and its `unknown @table argument` failure; `Ast.table_cfg` as
|
||
`{ table_name : string option; indexes : string list list }`; and the class
|
||
descriptor in `runtime/src/wob.h` that the loader validates. Adding a key is
|
||
genuinely small — the semantics are the iteration.
|
||
|
||
Forks the spec must settle:
|
||
|
||
1. **What are the modes called?** `ram` / `durable` / `cold` is descriptive of
|
||
mechanism. `scratch` / `persistent` / `archived` is descriptive of intent and
|
||
is what a developer reasons about. The names are the API and are hard to
|
||
change later; leaning the intent-shaped set for the first two if a
|
||
short-enough pair can be found, since a developer choosing a mode is thinking
|
||
about what the data is *for*, not about where it sits.
|
||
2. **How hard does the compiler push?** Three levels: warn on the suspicious
|
||
cases; refuse the provably-broken ones (a `durable`→`ram` `ref`); or a full
|
||
dataflow check that an insert into a `ram` table is never expected to persist.
|
||
The third is not statically decidable in general. Leaning: refuse the
|
||
relation case (provable, high value), warn on the `WO_DATA`-with-no-durable-
|
||
table case, and stop there.
|
||
3. **What is the default, and does it depend on iteration 1?** `durable` keeps
|
||
every existing program identical, which is nearly decisive. But if iteration
|
||
1's numbers show the WAL write dominating a workload nobody wanted durable,
|
||
there is an argument for making the choice mandatory — no default, every
|
||
`@table` states its mode. That is a bigger source change and a better
|
||
language; the fork is whether the churn is worth it now or at 1.0.
|
||
4. **Does `mode: ram` imply anything about the actor/DB-actor path?** Stage 3
|
||
marshals worker-shard statements to shard 0 because the owner shard holds the
|
||
store and the WAL. A `ram` table has no WAL — so does it still need to live on
|
||
the owner? A per-shard `ram` table would be dramatically faster and a
|
||
different consistency story. Tempting, out of scope here, and worth recording
|
||
as a candidate rather than deciding in passing.
|