docs(databasev2): third track — the database beyond RAM, with per-table storage modes
- docs/stories/databasev2/, numbered from 1. Six PENDING database iterations
moved from the language track and renumbered, keeping the old id in
`was_language_iteration:` so a search for "iteration 32" still finds it:
32 -> 3 WAL checkpoint, 23 -> 4 io_uring commit, 33 -> 7 single-file store,
27 -> 8 query grammar, 20 -> 9 cross-program, 21 -> 10 keypair auth.
Done work (9, 9b, 22) stays as v1 history; language 18 left whole
- the problem, read off the engine not guessed: rows are malloc'd slabs with
addresses stable forever, NO eviction/spill/paging anywhere in database/src,
the WAL never checkpoints so boot replays all history, and durability is one
process-global WO_DATA so no table can say it matters more than another.
An allocation failure IS a clean catchable WO_T_OOM — but swap thrash
arrives first and carries no error signal at all, which is the real hazard
- four new iterations:
1 measure the ceiling FIRST (curve not cliff; the three exits; kill -9 at
exhaustion) — every later default should follow from a number
2 `@table(mode: ram | durable | cold)` — the grammar ask. Small surface
(Ast.table_cfg gains a key, the parser already rejects unknown args), big
semantics: `durable` defaults so nothing changes silently, and the
compiler refuses a durable row holding a `ref` into a ram table
5 bounded tables + refuse/evict/back-pressure, shedding BEFORE the OS acts
6 cold tiering — mostly forks, incl. whether the language surfaces the
fault cost and whether @unique on cold is refused outright. A paged
B-tree stays rejected: if tiering needs one, reject tiering
- 39 links repointed, link TEXT renumbered to track-local ids; arc gains one
pointer row replacing the six moved; board + board-views cover three tracks
- linkcheck 0 broken / 0 anchors; no code blocks in any story
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
01df75245f
commit
746dc2b42b
21 changed files with 970 additions and 56 deletions
|
|
@ -2,8 +2,8 @@
|
||||||
|
|
||||||
> **Status: target workload — does not compile on today's toolchain.**
|
> **Status: target workload — does not compile on today's toolchain.**
|
||||||
> Written ahead of iterations
|
> Written ahead of iterations
|
||||||
> [20 (cross-program tables)](../../stories/language-runtime-database/20-cross-program-tables.md)
|
> [9 (cross-program tables)](../../stories/databasev2/09-cross-program-tables.md)
|
||||||
> and [21 (keypair attach auth)](../../stories/language-runtime-database/21-keypair-attach-auth.md),
|
> and [10 (keypair attach auth)](../../stories/databasev2/10-keypair-attach-auth.md),
|
||||||
> the way every acceptance sample here precedes its features. It also leans
|
> the way every acceptance sample here precedes its features. It also leans
|
||||||
> on 9/9b (the [employee sample](../employee/) it attaches to must run
|
> on 9/9b (the [employee sample](../employee/) it attaches to must run
|
||||||
> first).
|
> first).
|
||||||
|
|
|
||||||
|
|
@ -72,7 +72,7 @@ goes to find what took each one's place.
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `09-concurrency-scaleout.md` | [`08-shard-actor-runtime.md`](../stories/language-runtime-database/08-shard-actor-runtime.md) + [`11-fibers.md`](../stories/language-runtime-database/11-fibers.md) — the arc, landed 2026-08-21 |
|
| `09-concurrency-scaleout.md` | [`08-shard-actor-runtime.md`](../stories/language-runtime-database/08-shard-actor-runtime.md) + [`11-fibers.md`](../stories/language-runtime-database/11-fibers.md) — the arc, landed 2026-08-21 |
|
||||||
| `10-storage-foundations.md`, `11-wal-and-recovery.md` | [`09-database-engine.md`](../stories/language-runtime-database/09-database-engine.md) (typed WAL + replay) and [`22-durability-throughput-scale.md`](../stories/language-runtime-database/22-durability-throughput-scale.md) (the measurements) |
|
| `10-storage-foundations.md`, `11-wal-and-recovery.md` | [`09-database-engine.md`](../stories/language-runtime-database/09-database-engine.md) (typed WAL + replay) and [`22-durability-throughput-scale.md`](../stories/language-runtime-database/22-durability-throughput-scale.md) (the measurements) |
|
||||||
| `12-engine-disk-cutover.md` | Nothing — RAM stays authoritative by doctrine (principle 7). The disk story is the WAL; reclamation is [`32-wal-checkpoint.md`](../stories/language-runtime-database/32-wal-checkpoint.md) |
|
| `12-engine-disk-cutover.md` | Nothing — RAM stays authoritative by doctrine (principle 7). The disk story is the WAL; reclamation is [`databasev2 3, WAL checkpoint`](../stories/databasev2/03-wal-checkpoint.md) |
|
||||||
| `13-class-model-live-pricing.md` | [`09b-table-relations-query.md`](../stories/language-runtime-database/09b-table-relations-query.md) — `@table`, `ref`/`backlink`, the compiler-checked query surface |
|
| `13-class-model-live-pricing.md` | [`09b-table-relations-query.md`](../stories/language-runtime-database/09b-table-relations-query.md) — `@table`, `ref`/`backlink`, the compiler-checked query surface |
|
||||||
| `07-inotify-content-watcher.md` | [`07-logwatcher-proof.md`](../stories/language-runtime-database/07-logwatcher-proof.md) — the log-watcher sample polls via `fs.stat`; inotify was never surfaced as a builtin |
|
| `07-inotify-content-watcher.md` | [`07-logwatcher-proof.md`](../stories/language-runtime-database/07-logwatcher-proof.md) — the log-watcher sample polls via `fs.stat`; inotify was never surfaced as a builtin |
|
||||||
| `08-sendfile-static-assets.md` | Nothing. `sendfile` is not exposed; static assets are served as `Text` through `net.write` |
|
| `08-sendfile-static-assets.md` | Nothing. `sendfile` is not exposed; static assets are served as `Text` through `net.write` |
|
||||||
|
|
|
||||||
|
|
@ -46,7 +46,7 @@ The implementation phases that lean on this material:
|
||||||
- The Rust-era consumers (plans 10/11/12: storage foundations, WAL and
|
- The Rust-era consumers (plans 10/11/12: storage foundations, WAL and
|
||||||
recovery, disk cutover) were removed with that track 2026-08-18; their
|
recovery, disk cutover) were removed with that track 2026-08-18; their
|
||||||
ideas shipped in `database/src/` (typed WAL + replay) and the rest wait
|
ideas shipped in `database/src/` (typed WAL + replay) and the rest wait
|
||||||
on [iteration 32](../../../stories/language-runtime-database/32-wal-checkpoint.md)
|
on [databasev2 3](../../../stories/databasev2/03-wal-checkpoint.md)
|
||||||
(checkpoint) — the wal/buffer cards are its entry material.
|
(checkpoint) — the wal/buffer cards are its entry material.
|
||||||
- Current consumers: [constraints-and-grammar](constraints-and-grammar.md)
|
- Current consumers: [constraints-and-grammar](constraints-and-grammar.md)
|
||||||
(the `@table` PK/FK grammar direction) and
|
(the `@table` PK/FK grammar direction) and
|
||||||
|
|
|
||||||
|
|
@ -10,11 +10,14 @@ The single place to learn where this project stands. Organised in six buckets:
|
||||||
folders** — a doc stays where it was authored, and only its frontmatter, its
|
folders** — a doc stays where it was authored, and only its frontmatter, its
|
||||||
banner and this board change.
|
banner and this board change.
|
||||||
|
|
||||||
**Two tracks** (2026-08-26): [`language-runtime-database/`](language-runtime-database/00-story.md)
|
**Three tracks** (2026-08-26):
|
||||||
— the language, runtime and database — and [`porch/`](porch/00-story.md), the web
|
[`language-runtime-database/`](language-runtime-database/00-story.md) — the
|
||||||
framework written in it. Each numbers its iterations from 1, so a porch 3 is not
|
language and runtime; [`porch/`](porch/00-story.md) — the web framework written
|
||||||
a language 3; porch stories carry `track: porch` in frontmatter to keep queries
|
in it; [`databasev2/`](databasev2/00-story.md) — the database beyond RAM. Each
|
||||||
honest. Track folders are fine; **status** folders are not.
|
numbers its iterations from 1, so a porch 3 is not a language 3; every non-language
|
||||||
|
story carries `track:` in frontmatter, and moved ones keep
|
||||||
|
`was_language_iteration:` so a search for the old number still finds them. Track
|
||||||
|
folders are fine; **status** folders are not.
|
||||||
|
|
||||||
**Status lives in frontmatter, nowhere else** (directive 2026-08-26). Every
|
**Status lives in frontmatter, nowhere else** (directive 2026-08-26). Every
|
||||||
story iteration file sits flat in its track folder and
|
story iteration file sits flat in its track folder and
|
||||||
|
|
@ -348,19 +351,19 @@ that sequences its tasks. Read one, approve, then the next starts.
|
||||||
| 22 | [Durability, throughput, scale](language-runtime-database/22-durability-throughput-scale.md) | ✅ **landed 2026-08-21** — db-bench + baseline.json (74 metrics) + restart/kill -9 proofs both shard counts; durable 4.5k vs ram 297k inserts/s, reads O(table), msgrate 13.4M/2.45M |
|
| 22 | [Durability, throughput, scale](language-runtime-database/22-durability-throughput-scale.md) | ✅ **landed 2026-08-21** — db-bench + baseline.json (74 metrics) + restart/kill -9 proofs both shard counts; durable 4.5k vs ram 297k inserts/s, reads O(table), msgrate 13.4M/2.45M |
|
||||||
| 31 | [Actor lifecycle](language-runtime-database/31-actor-lifecycle.md) | 🔄 **absorbed into 24** (directive 2026-08-23) and half landed there: `call` request/response with a typed scalar reply (`WO_B_CALL = 88`, WO-E226), bounded mailboxes (`WO_MAILBOX`, cap 1024, catchable `WO_T_ACTOR`), and actor death that traps callers instead of hanging them. Still open: `monitor` and `time.after` — ids **89 and 90 are reserved holes** in `wob.h`, which is the machine-checkable proof of what is left. Supervision trees stay out of v1 |
|
| 31 | [Actor lifecycle](language-runtime-database/31-actor-lifecycle.md) | 🔄 **absorbed into 24** (directive 2026-08-23) and half landed there: `call` request/response with a typed scalar reply (`WO_B_CALL = 88`, WO-E226), bounded mailboxes (`WO_MAILBOX`, cap 1024, catchable `WO_T_ACTOR`), and actor death that traps callers instead of hanging them. Still open: `monitor` and `time.after` — ids **89 and 90 are reserved holes** in `wob.h`, which is the machine-checkable proof of what is left. Supervision trees stay out of v1 |
|
||||||
| 24 | [chat: WebSocket workload](language-runtime-database/24-chat-websocket-workload.md) | 🔄 **the live slice** (absorbing 31 + 34, directive 2026-08-23) — branch `chat-ws-lifecycle`, 5/10 tasks landed: crypto, bounded mailboxes, WS upgrade, frame codec, `call`/reply + actor death. Pending: `monitor`, `time.after`, the chat sample, its gate, closeout. State lives in [the marker](../active-slice-2026-08-23-chat-ws-lifecycle.md) |
|
| 24 | [chat: WebSocket workload](language-runtime-database/24-chat-websocket-workload.md) | 🔄 **the live slice** (absorbing 31 + 34, directive 2026-08-23) — branch `chat-ws-lifecycle`, 5/10 tasks landed: crypto, bounded mailboxes, WS upgrade, frame codec, `call`/reply + actor death. Pending: `monitor`, `time.after`, the chat sample, its gate, closeout. State lives in [the marker](../active-slice-2026-08-23-chat-ws-lifecycle.md) |
|
||||||
| 23 | [io_uring group-commit](language-runtime-database/23-io-uring-commit.md) | ⬜ fifth in chain, after stage 3 + 22 |
|
| 23 | [io_uring group-commit](databasev2/04-io-uring-commit.md) | ⬜ fifth in chain, after stage 3 + 22 |
|
||||||
| 32 | [WAL checkpoint](language-runtime-database/32-wal-checkpoint.md) | ⬜ last in chain, after 23 — disk reclamation + bounded replay (story written 2026-08-21) |
|
| 32 | [WAL checkpoint](databasev2/03-wal-checkpoint.md) | ⬜ last in chain, after 23 — disk reclamation + bounded replay (story written 2026-08-21) |
|
||||||
| 33 | [Single-file store](language-runtime-database/33-single-file-db.md) | ⬜ off-chain, small — `WO_DATA=<path>.db` file form; driver-only (story written 2026-08-22) |
|
| 33 | [Single-file store](databasev2/07-single-file-db.md) | ⬜ off-chain, small — `WO_DATA=<path>.db` file form; driver-only (story written 2026-08-22) |
|
||||||
| 34 | [Crypto builtins](language-runtime-database/34-crypto-builtins.md) | 🔄 **code landed** as 24's T1 (`d14fa9f`): `sha1`/`sha256`/`hmac_sha256`, ids 85–87 in `wob.h`, `runtime/src/crypto.c`, RFC/FIPS vectors 18/0, corpus pin. The 24 gate that once needed it is cleared. Frontmatter keeps `status: refine` only until 24's T10 closeout sets it to `done` |
|
| 34 | [Crypto builtins](language-runtime-database/34-crypto-builtins.md) | 🔄 **code landed** as 24's T1 (`d14fa9f`): `sha1`/`sha256`/`hmac_sha256`, ids 85–87 in `wob.h`, `runtime/src/crypto.c`, RFC/FIPS vectors 18/0, corpus pin. The 24 gate that once needed it is cleared. Frontmatter keeps `status: refine` only until 24's T10 closeout sets it to `done` |
|
||||||
| 38 | [Content platform capabilities](language-runtime-database/38-content-platform-capabilities.md) | ⬜ off-chain, needs a spec — the two capability families no iteration owns, confirmed against `runtime/src/wob.h`: `fs` mutation verbs (six fs builtins, ids 40–45; `append` creates-if-absent, so nothing is ever replaced, truncated, deleted or renamed) and `net.connect` (ids 51–55 + 91–95, no connect, and no `connect()` anywhere in `runtime/src/` — so no OIDC/SMTP/object-store/webhook/federation). Driven by a `docs/examples/vault` content-collaboration workload, in 28's mould. New builtins from 96 (89/90 reserved for 31); no `.wob` bump (`WOB_VERSION 6u`, last moved by 36). Story written 2026-08-26 from the "can it build a Nextcloud?" ask |
|
| 38 | [Content platform capabilities](language-runtime-database/38-content-platform-capabilities.md) | ⬜ off-chain, needs a spec — the two capability families no iteration owns, confirmed against `runtime/src/wob.h`: `fs` mutation verbs (six fs builtins, ids 40–45; `append` creates-if-absent, so nothing is ever replaced, truncated, deleted or renamed) and `net.connect` (ids 51–55 + 91–95, no connect, and no `connect()` anywhere in `runtime/src/` — so no OIDC/SMTP/object-store/webhook/federation). Driven by a `docs/examples/vault` content-collaboration workload, in 28's mould. New builtins from 96 (89/90 reserved for 31); no `.wob` bump (`WOB_VERSION 6u`, last moved by 36). Story written 2026-08-26 from the "can it build a Nextcloud?" ask |
|
||||||
| 39 | [Web framework parity](language-runtime-database/39-web-framework-parity.md) | ⬜ off-chain, needs a spec — from [the Fiber v3.5.0 study](../plan/exploration/fiber/00-fiber-parity.md) (all 32 of its middleware read against `porch`; **nine already have a counterpart**). Leads with a **random-bytes builtin**: the framework ledger claimed CSRF/sessions were unblocked by iteration 34's HMAC, but HMAC authenticates a token and cannot mint one — there is no RNG anywhere in the runtime. Then cookies (absent both ways; `Resp.headers` being a map cannot carry two `Set-Cookie` lines), then limiter/idempotency (cheapest wins — `@table` + `time.ticks`, nothing new), sessions, CSRF, and the routing/response sugar. Streaming/SSE/compression, `@derive` binding, TTL cache, `proxy` and metrics all excluded with owners named |
|
| 39 | [Web framework parity](language-runtime-database/39-web-framework-parity.md) | ⬜ off-chain, needs a spec — from [the Fiber v3.5.0 study](../plan/exploration/fiber/00-fiber-parity.md) (all 32 of its middleware read against `porch`; **nine already have a counterpart**). Leads with a **random-bytes builtin**: the framework ledger claimed CSRF/sessions were unblocked by iteration 34's HMAC, but HMAC authenticates a token and cannot mint one — there is no RNG anywhere in the runtime. Then cookies (absent both ways; `Resp.headers` being a map cannot carry two `Set-Cookie` lines), then limiter/idempotency (cheapest wins — `@table` + `time.ticks`, nothing new), sessions, CSRF, and the routing/response sugar. Streaming/SSE/compression, `@derive` binding, TTL cache, `proxy` and metrics all excluded with owners named |
|
||||||
| 37 | [wo-html components](language-runtime-database/37-wo-html-components.md) | ✅ off-chain — LANDED 2026-08-25. Raw text literal (backtick, margin stripped at lex time, `{{ }}` auto-escapes) + the component layer: `Component`/`render_all`/`Layout` in wo-html, `ok_html` moved into the framework, site and shop both migrated |
|
| 37 | [wo-html components](language-runtime-database/37-wo-html-components.md) | ✅ off-chain — LANDED 2026-08-25. Raw text literal (backtick, margin stripped at lex time, `{{ }}` auto-escapes) + the component layer: `Component`/`render_all`/`Layout` in wo-html, `ok_html` moved into the framework, site and shop both migrated |
|
||||||
| 35 | [net runtime seams](language-runtime-database/35-net-runtime-seams.md) | ⬜ off-chain — fd deadlines on the park plane, Unix sockets, peer address; owns the ledger's three 🔧 rows (story written 2026-08-22) |
|
| 35 | [net runtime seams](language-runtime-database/35-net-runtime-seams.md) | ⬜ off-chain — fd deadlines on the park plane, Unix sockets, peer address; owns the ledger's three 🔧 rows (story written 2026-08-22) |
|
||||||
| 20 | [Cross-program tables](language-runtime-database/20-cross-program-tables.md) | ⏸ hold (2026-08-21); channel done (branch ipc-attach keeps its manifest) |
|
| 20 | [Cross-program tables](databasev2/09-cross-program-tables.md) | ⏸ hold (2026-08-21); channel done (branch ipc-attach keeps its manifest) |
|
||||||
| 21 | [Keypair attach auth](language-runtime-database/21-keypair-attach-auth.md) | ⏸ hold (2026-08-21); crypto+handshake done (branch keypair-auth keeps its manifest) |
|
| 21 | [Keypair attach auth](databasev2/10-keypair-attach-auth.md) | ⏸ hold (2026-08-21); crypto+handshake done (branch keypair-auth keeps its manifest) |
|
||||||
| 25 | [HTTP service layer](../superpowers/plans/2026-08-01-http-service-layer.md) | ⏸ hold (2026-08-21) — story file removed; the plan doc remains |
|
| 25 | [HTTP service layer](../superpowers/plans/2026-08-01-http-service-layer.md) | ⏸ hold (2026-08-21) — story file removed; the plan doc remains |
|
||||||
| 26 | [Blue-green deploy](language-runtime-database/26-blue-green-deploy.md) | ⏸ hold (2026-08-21) |
|
| 26 | [Blue-green deploy](language-runtime-database/26-blue-green-deploy.md) | ⏸ hold (2026-08-21) |
|
||||||
| 27 | [Query grammar corpus](language-runtime-database/27-query-grammar-corpus.md) | ⏸ hold (2026-08-21) |
|
| 27 | [Query grammar corpus](databasev2/08-query-grammar-corpus.md) | ⏸ hold (2026-08-21) |
|
||||||
| 28 | [skillhost host workload](language-runtime-database/28-skillhost-host-workload.md) | ⏸ hold (2026-08-21); gaps recorded (branch query-grammar found skillhost needs no new query grammar) |
|
| 28 | [skillhost host workload](language-runtime-database/28-skillhost-host-workload.md) | ⏸ hold (2026-08-21); gaps recorded (branch query-grammar found skillhost needs no new query grammar) |
|
||||||
| 29 | [Compile-time metaprogramming](language-runtime-database/29-compile-time-metaprogramming.md) | ⏸ hold (2026-08-21) |
|
| 29 | [Compile-time metaprogramming](language-runtime-database/29-compile-time-metaprogramming.md) | ⏸ hold (2026-08-21) |
|
||||||
| 15 | [deps: `wo.toml [deps]`](language-runtime-database/15-deps-package-manager.md) | ✅ **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 |
|
| 15 | [deps: `wo.toml [deps]`](language-runtime-database/15-deps-package-manager.md) | ✅ **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 |
|
||||||
|
|
@ -591,7 +594,7 @@ precedence notes for resumption.
|
||||||
5. **23** — io_uring group-commit; the WAL's WRITE+FSYNC chains ride the
|
5. **23** — io_uring group-commit; the WAL's WRITE+FSYNC chains ride the
|
||||||
arc's per-shard ring (T4); after 22's baseline — the payoff, measured.
|
arc's per-shard ring (T4); after 22's baseline — the payoff, measured.
|
||||||
6. **32** — WAL checkpoint
|
6. **32** — WAL checkpoint
|
||||||
([story](language-runtime-database/32-wal-checkpoint.md),
|
([story](databasev2/03-wal-checkpoint.md),
|
||||||
written 2026-08-21): the WAL is append-only forever — snapshot +
|
written 2026-08-21): the WAL is append-only forever — snapshot +
|
||||||
truncate reclaims disk and bounds replay; after 23 (composes with
|
truncate reclaims disk and bounds replay; after 23 (composes with
|
||||||
group-commit), policy set by 22's aged-store numbers.
|
group-commit), policy set by 22's aged-store numbers.
|
||||||
|
|
@ -599,6 +602,36 @@ precedence notes for resumption.
|
||||||
**30** — observability, CI, fuzz: named 2026-08-20, still row-only (no
|
**30** — observability, CI, fuzz: named 2026-08-20, still row-only (no
|
||||||
story file); slots in when scheduled — nothing in the chain depends on it.
|
story file); slots in when scheduled — nothing in the chain depends on it.
|
||||||
|
|
||||||
|
### ▸ databasev2 — the database beyond RAM
|
||||||
|
|
||||||
|
New 2026-08-26. **The problem:** RAM is authoritative (principle 7) and nothing
|
||||||
|
declares a budget. Rows live in `malloc`'d slabs whose addresses are stable
|
||||||
|
forever; there is no eviction, spill or paging anywhere in `database/src/`; the
|
||||||
|
WAL never checkpoints so boot replays all history; and durability is one
|
||||||
|
process-global `WO_DATA`, so no table can say it matters more than another. An
|
||||||
|
allocation failure *is* a clean catchable `WO_T_OOM` — but swap thrash arrives
|
||||||
|
first and carries no error signal at all.
|
||||||
|
|
||||||
|
**The lever** is per-table storage modes, which is why this track has a grammar
|
||||||
|
iteration. Six pending iterations moved here from the language track (their old
|
||||||
|
ids in the rows below); four are new. Done database work — 9, 9b, 22 — stays in
|
||||||
|
the language arc as v1 history.
|
||||||
|
|
||||||
|
| # | Iteration | State |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 1 | [RAM ceiling: measure the breaking point](databasev2/01-ram-ceiling-measurement.md) | ⬜ **first, and startable today** — nobody here can say what happens at 90% RAM. Curve not cliff: swap onset, latency departure, the three exits (checked trap / swap thrash / OOM killer), and `kill -9` durability *at exhaustion*. Output is `perf-targets.md` + baseline rows, not prose |
|
||||||
|
| 2 | [`@table` storage modes](databasev2/02-table-storage-modes.md) | ⬜ **the language enrichment** — `mode: ram \| durable \| cold` per table, replacing the global switch. `durable` defaults so nothing changes silently; the compiler refuses a `durable` row holding a `ref` into a `ram` table. `.wob` format change. Grammar is small (`Ast.table_cfg` gains a key); semantics are the iteration |
|
||||||
|
| 3 | [WAL checkpoint](databasev2/03-wal-checkpoint.md) *(was 32)* | ⬜ snapshot + truncate: disk reclaimed, replay bounded |
|
||||||
|
| 4 | [io_uring group commit](databasev2/04-io-uring-commit.md) *(was 23)* | ⬜ close the 66× gap iteration 22 measured (durable 4.5k vs ram 297k inserts/s) |
|
||||||
|
| 5 | [Bounded tables and eviction](databasev2/05-bounded-tables-eviction.md) | ⬜ a declared capacity + refuse/evict/back-pressure, and a process-level pressure signal that sheds **before** the allocator or OS gets involved — turning the invisible failure into a managed one |
|
||||||
|
| 6 | [Cold tiering](databasev2/06-cold-tiering.md) | ⬜ the iteration that raises the ceiling, and the riskiest. Mostly forks: which shape, whether the index itself fits, whether the *language* surfaces the fault cost, and whether `@unique` on a cold table is refused outright. A paged B-tree stays rejected — if tiering needs one, reject tiering |
|
||||||
|
| 7 | [Single-file store](databasev2/07-single-file-db.md) *(was 33)* | ⬜ `WO_DATA=<path>.db`; driver-only, independent |
|
||||||
|
| 8 | [Query grammar from corpora](databasev2/08-query-grammar-corpus.md) *(was 27)* | ⬜ whole-query `count`, `exists`; independent |
|
||||||
|
| 9 | [Cross-program tables](databasev2/09-cross-program-tables.md) *(was 20)* | ⏸ hold — attach to a running program's database over local IPC |
|
||||||
|
| 10 | [Keypair attach auth](databasev2/10-keypair-attach-auth.md) *(was 21)* | ⏸ hold — program identity as a keypair; needs 9 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### ▸ porch — the web framework track
|
### ▸ porch — the web framework track
|
||||||
|
|
||||||
New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md).
|
New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md).
|
||||||
|
|
|
||||||
|
|
@ -28,12 +28,13 @@ standup narrative; these queries are the live views over the same facts.
|
||||||
Adjust the `FROM` path to your vault root (queries below assume the
|
Adjust the `FROM` path to your vault root (queries below assume the
|
||||||
vault opens at the repo root).
|
vault opens at the repo root).
|
||||||
|
|
||||||
Two tracks now carry iterations, each numbered from 1:
|
Three tracks now carry iterations, each numbered from 1:
|
||||||
`language-runtime-database/` (the language, runtime and database) and `porch/`
|
`language-runtime-database/` (the language and runtime), `porch/` (the web
|
||||||
(the web framework, added 2026-08-26). Iteration ids therefore repeat across
|
framework) and `databasev2/` (the database beyond RAM) — the latter two added
|
||||||
tracks — a porch 3 is not a language 3 — so every query below is scoped by
|
2026-08-26. Iteration ids therefore repeat across tracks, so every query below is
|
||||||
`FROM` path, and porch stories carry `track: porch` so a combined query can
|
scoped by `FROM` path; non-language stories carry `track:`, and iterations moved
|
||||||
still tell them apart.
|
between tracks keep `was_language_iteration:` so the old number stays
|
||||||
|
searchable.
|
||||||
|
|
||||||
## Everything not done, chain order first
|
## Everything not done, chain order first
|
||||||
|
|
||||||
|
|
@ -79,6 +80,15 @@ the place status is edited.** A status change is one edit to one
|
||||||
`status:` key; a Kanban card drag that only rewrites the Kanban file is a
|
`status:` key; a Kanban card drag that only rewrites the Kanban file is a
|
||||||
lie the next query won't see.
|
lie the next query won't see.
|
||||||
|
|
||||||
|
## The databasev2 track
|
||||||
|
|
||||||
|
```dataview
|
||||||
|
TABLE iteration, status, was_language_iteration AS "was"
|
||||||
|
FROM "docs/stories/databasev2"
|
||||||
|
WHERE status != "done"
|
||||||
|
SORT iteration ASC
|
||||||
|
```
|
||||||
|
|
||||||
## The porch track
|
## The porch track
|
||||||
|
|
||||||
```dataview
|
```dataview
|
||||||
|
|
|
||||||
152
docs/stories/databasev2/00-story.md
Normal file
152
docs/stories/databasev2/00-story.md
Normal file
|
|
@ -0,0 +1,152 @@
|
||||||
|
# Story — databasev2: the database beyond RAM
|
||||||
|
|
||||||
|
The third track. `language-runtime-database/` built the engine;
|
||||||
|
[`porch/`](../porch/00-story.md) is the framework on top; this track answers the
|
||||||
|
question v1 deliberately deferred: **what happens when the data does not fit in
|
||||||
|
memory.**
|
||||||
|
|
||||||
|
Numbering restarts at 1, local to this track. Frontmatter carries
|
||||||
|
`track: databasev2`, and iterations moved here keep their old id in
|
||||||
|
`was_language_iteration:` so a search for "iteration 32" still finds the WAL
|
||||||
|
checkpoint. Status stays where it belongs — the `status:` key, never a directory.
|
||||||
|
|
||||||
|
## The problem, stated honestly
|
||||||
|
|
||||||
|
Principle 7 says **RAM is authoritative; the WAL makes it durable.** That is a
|
||||||
|
real design, not a shortcut: reads never touch disk, so latency is predictable,
|
||||||
|
and durability is a sequential append rather than a storage engine bolted to the
|
||||||
|
side. Iteration 22 measured what it buys — reads at 1.3M ops/s after the index
|
||||||
|
probe landed, p50 1µs.
|
||||||
|
|
||||||
|
The bill comes due at the ceiling. Read from the engine as it stands:
|
||||||
|
|
||||||
|
- **Rows live in `malloc`'d slabs of 256, and their addresses are stable
|
||||||
|
forever** (`database/src/table.c`, `DB_SLAB_ROWS`). Slabs are allocated as a
|
||||||
|
table grows and freed only when the table is destroyed. The free-slot list
|
||||||
|
recycles removed slots, so a delete-heavy table plateaus — but a growing table
|
||||||
|
only grows.
|
||||||
|
- **The ceiling is process RSS, not a configured number.** `WO_HEAP_MB` (default
|
||||||
|
64 MiB) bounds the VM object arena; table storage is separate `malloc`, so
|
||||||
|
nothing in the system declares a maximum dataset size. There is no knob that
|
||||||
|
says "this database may use at most N".
|
||||||
|
- **There is no eviction, no spill, no paging, no LRU.** Grep
|
||||||
|
`database/src/` for any of them and nothing comes back. Every row ever
|
||||||
|
inserted and not deleted is resident.
|
||||||
|
- **The WAL is append-only with no checkpoint.** Boot replays every record ever
|
||||||
|
written, so startup time is O(all writes in the file's history) and disk grows
|
||||||
|
without bound. That is databasev2 [3](03-wal-checkpoint.md).
|
||||||
|
- **Durability is process-global.** `WO_DATA` is one environment variable that
|
||||||
|
turns on one `shard-0.wal` for the whole process (`runtime/src/main.c`). There
|
||||||
|
is no way to say "this table matters, that one is scratch".
|
||||||
|
|
||||||
|
### What actually breaks first
|
||||||
|
|
||||||
|
Worth being precise, because the failure mode determines the fix — and the good
|
||||||
|
news is that the engine's own behaviour is clean:
|
||||||
|
|
||||||
|
**An allocation failure is a catchable trap, not a crash.** Every `malloc` in
|
||||||
|
the row encoder is checked and jumps to an `oom` label; `DB_ERR_OOM` maps to
|
||||||
|
`WO_T_OOM`, which a program can `try`/`catch`. So a writeonce program that runs
|
||||||
|
out of memory *refuses the insert* rather than corrupting or dying. That is a
|
||||||
|
much better starting position than most engines have.
|
||||||
|
|
||||||
|
**But the trap is almost never what a real deployment hits first.** Long before
|
||||||
|
`malloc` returns NULL, the box starts swapping, and a RAM-authoritative database
|
||||||
|
on swap is the worst of both worlds: it has paid for in-memory data structures
|
||||||
|
and is now serving them from disk with no read path designed for that. On a
|
||||||
|
cgroup-limited host the OOM killer arrives instead, and an external `SIGKILL` is
|
||||||
|
the one shutdown path that skips every guarantee the WAL was written to provide —
|
||||||
|
though ack-after-fsync means acked writes still survive; iteration 22's `kill -9`
|
||||||
|
battery proves that much.
|
||||||
|
|
||||||
|
So the honest problem statement is not "malloc fails". It is: **there is no
|
||||||
|
declared budget, no back-pressure as the budget is approached, and no way to
|
||||||
|
distinguish data that must be resident from data that merely is.** Iteration
|
||||||
|
[1](01-ram-ceiling-measurement.md) exists to replace this paragraph with
|
||||||
|
numbers before anything is designed on top of it.
|
||||||
|
|
||||||
|
## The lever: per-table storage modes
|
||||||
|
|
||||||
|
The developer's ask, and the reason this track has a grammar iteration.
|
||||||
|
|
||||||
|
Today every `@table` is identical: resident, and durable if and only if
|
||||||
|
`WO_DATA` is set for the whole process. Real applications are not uniform —
|
||||||
|
a session table, a rate-limit counter and a page cache want *resident and
|
||||||
|
disposable*; an orders table wants *resident and durable*; an audit log wants
|
||||||
|
*durable and rarely read*. One global switch cannot express that, so it forces
|
||||||
|
either "everything is precious" or "nothing is".
|
||||||
|
|
||||||
|
Extending `@table` with a storage mode moves the decision into the language,
|
||||||
|
where the compiler can act on it:
|
||||||
|
|
||||||
|
- **`ram`** — resident, never WAL-logged, gone on restart. The compiler knows
|
||||||
|
no durability code is needed; the engine knows these rows are the first
|
||||||
|
candidates to shed under pressure; and — the part that matters — a program
|
||||||
|
that expects a `ram` table to survive a restart is now stating something the
|
||||||
|
compiler can refuse.
|
||||||
|
- **`durable`** — today's behaviour: resident and WAL-logged, ack after fsync.
|
||||||
|
- **`cold`** — durable, and *not* required to be resident. This is the mode that
|
||||||
|
actually raises the ceiling, and it is the one with real design work behind it
|
||||||
|
(iteration [6](06-cold-tiering.md)).
|
||||||
|
|
||||||
|
The grammar change is small and the surface is already the right shape:
|
||||||
|
`Ast.table_cfg` is `{ table_name; indexes }`, the parser's argument match
|
||||||
|
already rejects unknown keys with a catalogued diagnostic
|
||||||
|
(`unknown @table argument ... (supported: name, index)`), and adding one more
|
||||||
|
key follows the path `index` already cut. The *semantics* are the work, not the
|
||||||
|
syntax — which is exactly why it gets its own iteration
|
||||||
|
([2](02-table-storage-modes.md)) and why it comes after the measurement.
|
||||||
|
|
||||||
|
This is also the honest answer to "does this break principle 7?" It does not.
|
||||||
|
RAM stays authoritative **for the tables that say so**. `cold` is a declared
|
||||||
|
exception a developer opts into per table, with the trade written at the
|
||||||
|
declaration site rather than buried in an operations runbook.
|
||||||
|
|
||||||
|
## The sequence
|
||||||
|
|
||||||
|
| # | Iteration | Delivers | Needs |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 1 | [RAM ceiling: measure the breaking point](01-ram-ceiling-measurement.md) | what actually happens from 50% RAM to OOM — swap onset, latency cliff, trap behaviour, `kill -9` survival | nothing; extends iteration 22's harness |
|
||||||
|
| 2 | [`@table` storage modes](02-table-storage-modes.md) | the grammar: `mode: ram \| durable \| cold`, per table, replacing the global `WO_DATA` all-or-nothing | 1 for its defaults |
|
||||||
|
| 3 | [WAL checkpoint](03-wal-checkpoint.md) *(was language 32)* | snapshot + truncate: disk reclaimed, replay bounded | 4 composes |
|
||||||
|
| 4 | [io_uring group commit](04-io-uring-commit.md) *(was language 23)* | close the 66× durable/RAM write gap (4.5k vs 297k inserts/s) | the arc (landed) |
|
||||||
|
| 5 | [Bounded tables and eviction](05-bounded-tables-eviction.md) | a capacity a `ram` table may not exceed, and what happens when it does | 2 |
|
||||||
|
| 6 | [Cold tiering](06-cold-tiering.md) | rows that leave RAM and come back — the iteration that raises the ceiling | 2, 3, 5 |
|
||||||
|
| 7 | [Single-file store](07-single-file-db.md) *(was language 33)* | `WO_DATA=<path>.db` — a file path IS the store | independent |
|
||||||
|
| 8 | [Query grammar from corpora](08-query-grammar-corpus.md) *(was language 27)* | whole-query `count`, `exists` | independent |
|
||||||
|
| 9 | [Cross-program tables](09-cross-program-tables.md) *(was language 20)* | attach to a running program's database over local IPC | independent |
|
||||||
|
| 10 | [Keypair attach auth](10-keypair-attach-auth.md) *(was language 21)* | program identity as a keypair; mutual challenge–response | 9 |
|
||||||
|
|
||||||
|
```
|
||||||
|
1 ──▶ 2 ──▶ 5 ──▶ 6
|
||||||
|
│ ▲
|
||||||
|
3 ──▶ 4 ─────┘
|
||||||
|
7, 8 independent
|
||||||
|
9 ──▶ 10
|
||||||
|
```
|
||||||
|
|
||||||
|
Order rationale: **1 before 2** because a mode's default should follow from a
|
||||||
|
measurement, not a guess. **3 and 4 before 6** because tiering onto a log that
|
||||||
|
never truncates would make the disk problem worse, not better. **5 before 6**
|
||||||
|
because eviction from a bounded resident table is the simpler half of the same
|
||||||
|
mechanism, and getting the policy right there de-risks the hard half.
|
||||||
|
|
||||||
|
## What this track does NOT own
|
||||||
|
|
||||||
|
| Not databasev2's | Owner |
|
||||||
|
| --- | --- |
|
||||||
|
| `transaction { }` and `@table` feature flags | language [iteration 18](../language-runtime-database/18-memory-db-features.md) — approved spec, left whole on purpose |
|
||||||
|
| the TTL cache middleware | also language 18 (and [porch 1](../porch/01-store-backed-middleware.md) points there) |
|
||||||
|
| typed binding of rows into app classes | language [iteration 29 `@derive`](../language-runtime-database/29-compile-time-metaprogramming.md) |
|
||||||
|
| `fs` mutation verbs, outbound sockets | language [iteration 38](../language-runtime-database/38-content-platform-capabilities.md) |
|
||||||
|
| benchmark harness and CI | iteration 22 (landed) built the harness; per-change CI is language iteration 30 |
|
||||||
|
| a paged B-tree storage engine | **nobody, deliberately.** Recorded as rejected in [`discarded.md`](../../plan/discarded.md): the disk story is the WAL. `cold` tiering is not a licence to build SQLite. |
|
||||||
|
|
||||||
|
## Review protocol
|
||||||
|
|
||||||
|
The language track's, unchanged: one iteration read and approved before the next
|
||||||
|
starts; every iteration an unsplittable slice with phases, per-phase tasks,
|
||||||
|
Given/When/Then criteria and an out-of-scope list. Every engine change is gated
|
||||||
|
by `just employee`, `just db-actor` and `just db-bench` against
|
||||||
|
`bench/baseline.json` — and any iteration that claims a performance change must
|
||||||
|
move a number in that baseline, or it did not happen.
|
||||||
148
docs/stories/databasev2/01-ram-ceiling-measurement.md
Normal file
148
docs/stories/databasev2/01-ram-ceiling-measurement.md
Normal file
|
|
@ -0,0 +1,148 @@
|
||||||
|
---
|
||||||
|
track: databasev2
|
||||||
|
iteration: "1"
|
||||||
|
status: refine
|
||||||
|
---
|
||||||
|
|
||||||
|
# databasev2 1 — the RAM ceiling: measure the breaking point before designing for it
|
||||||
|
|
||||||
|
> Part of [Story — databasev2: the database beyond RAM](00-story.md).
|
||||||
|
>
|
||||||
|
> **First because the repo's own doctrine says so.** "Always inspect crashsites.
|
||||||
|
> Always measure. Never assume." Every later iteration in this track — the
|
||||||
|
> storage modes' defaults, the eviction policy, the tiering threshold — is a
|
||||||
|
> decision that should follow from a number. Right now nobody in this project
|
||||||
|
> can say what happens to a writeonce program at 90% of RAM, and designing
|
||||||
|
> tiering without that is guessing with extra steps.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- **Find the curve, not the cliff.** Not "does it die" — it dies, everything
|
||||||
|
does. What matters is the shape on the way down: at what fraction of RAM does
|
||||||
|
p99 read latency leave its 1µs baseline, what does insert throughput do as
|
||||||
|
slabs stop coming from a warm allocator, and how much warning is there between
|
||||||
|
"fine" and "unusable".
|
||||||
|
- **Characterise all three exits.** The engine can leave the happy path three
|
||||||
|
ways and they are not equally survivable: a checked `malloc` failure
|
||||||
|
(`DB_ERR_OOM` → `WO_T_OOM`, a catchable trap — the clean one), swap thrash
|
||||||
|
(no trap, no error, just latency collapse — the dangerous one because nothing
|
||||||
|
reports it), and the external OOM killer (`SIGKILL`, skipping every shutdown
|
||||||
|
path). Establish which arrives first under realistic limits, because the
|
||||||
|
answer determines whether the fix is back-pressure or eviction.
|
||||||
|
- **Prove the durability floor holds at the ceiling.** Iteration 22's `kill -9`
|
||||||
|
battery proved acked writes survive under load. Re-run it *at memory
|
||||||
|
exhaustion*, which is a different and nastier state — an allocation failure
|
||||||
|
mid-commit is exactly where an ack-before-durable bug would hide.
|
||||||
|
- **Publish numbers others can build on.** The output is a section in
|
||||||
|
`perf-targets.md` and rows in `bench/baseline.json`, not a paragraph of
|
||||||
|
prose. A measurement that only printed once is not a measurement.
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase A — a workload that can actually reach the ceiling
|
||||||
|
|
||||||
|
- Extend `docs/examples/db-bench` with a growth mode: insert until a target RSS
|
||||||
|
fraction, holding row shape and index count constant so the variable is size
|
||||||
|
alone.
|
||||||
|
- Run it under an explicit memory limit (a cgroup or `ulimit`) rather than on a
|
||||||
|
big box — "it survived on a 64 GB workstation" measures the workstation.
|
||||||
|
- Record RSS against row count so the per-row overhead is known: slab headroom,
|
||||||
|
the id hash, the secondary-index multimaps and the per-row engine-owned values
|
||||||
|
(`db_text`, `db_rec`, `db_multi`, `db_map` are each their own allocation).
|
||||||
|
- Verify: RSS growth is linear and its slope is written down; the run is
|
||||||
|
reproducible twice within the tolerance policy iteration 22 established.
|
||||||
|
|
||||||
|
### Phase B — the latency and throughput curve
|
||||||
|
|
||||||
|
- Sample read p50/p99, query p99 and insert throughput at fixed fractions of the
|
||||||
|
limit, so the result is a curve rather than two endpoints.
|
||||||
|
- Separate the two effects deliberately: allocator pressure (still resident) and
|
||||||
|
swap (no longer resident). They have different fixes and conflating them would
|
||||||
|
send iteration 6 after the wrong one.
|
||||||
|
- Include the DB-actor path, since a cross-shard statement's reply materialises
|
||||||
|
a copy — memory pressure and the actor RPC interact and nobody has looked.
|
||||||
|
- Verify: the curve is recorded per metric class with iteration 22's per-class
|
||||||
|
tolerances; the swap onset point is identified, not interpolated.
|
||||||
|
|
||||||
|
### Phase C — the three exits, deliberately triggered
|
||||||
|
|
||||||
|
- Drive a checked allocation failure and confirm `WO_T_OOM` is catchable, the
|
||||||
|
insert is refused whole, no partial row or index entry is left, and the
|
||||||
|
process continues serving.
|
||||||
|
- Drive swap thrash and record what a client sees. This is the case with no
|
||||||
|
error signal at all, and naming it is most of the value of this iteration.
|
||||||
|
- Drive the OOM killer under a cgroup limit and confirm what survives: replay
|
||||||
|
the WAL and check every acked write is present.
|
||||||
|
- Verify: the trap path leaves no torn state (row count and index agree after a
|
||||||
|
refused insert); replay after `SIGKILL` at exhaustion loses no acked write.
|
||||||
|
|
||||||
|
### Phase D — write it down where decisions get made
|
||||||
|
|
||||||
|
- A `perf-targets.md` section with the curve, the swap onset, the per-row
|
||||||
|
overhead and the exit characterisation.
|
||||||
|
- Baseline rows for the growth metrics so a regression is caught by the existing
|
||||||
|
gate rather than by a person remembering.
|
||||||
|
- A short statement of what the numbers *imply* for iterations 2, 5 and 6 —
|
||||||
|
which is the point of going first.
|
||||||
|
- Verify: `just db-bench` green against the extended baseline; the gate bites
|
||||||
|
when a growth metric is doctored.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- **Given** the growth workload under a fixed memory limit, **when** it runs
|
||||||
|
twice, **then** RSS-per-row agrees within the tolerance policy and the slope
|
||||||
|
is recorded in `perf-targets.md`.
|
||||||
|
- **Given** the workload at rising RAM fractions, **when** latency is sampled,
|
||||||
|
**then** the fraction at which read p99 first leaves its baseline is
|
||||||
|
identified as a measured point, not an estimate.
|
||||||
|
- **Given** a deliberately induced allocation failure, **when** an insert is
|
||||||
|
attempted, **then** it traps `WO_T_OOM` catchably, the table's row count is
|
||||||
|
unchanged, every index agrees with the slab contents, and the process keeps
|
||||||
|
serving subsequent requests.
|
||||||
|
- **Given** swap thrash, **when** a client issues reads, **then** the observed
|
||||||
|
degradation is quantified and the fact that **no error is surfaced** is
|
||||||
|
recorded explicitly as a finding.
|
||||||
|
- **Given** a cgroup limit and a workload that exceeds it, **when** the OOM
|
||||||
|
killer fires, **then** replaying the WAL shows every acked write present —
|
||||||
|
ack-after-fsync holding in the one shutdown path that skips all cleanup.
|
||||||
|
- **Given** the extended baseline, **when** a growth metric is doctored, **then**
|
||||||
|
`just db-bench` fails on exactly that metric.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- **Any fix.** This iteration measures. Eviction is
|
||||||
|
[5](05-bounded-tables-eviction.md), tiering is [6](06-cold-tiering.md),
|
||||||
|
declared budgets are [2](02-table-storage-modes.md). Shipping a fix inside the
|
||||||
|
measurement slice would remove the ability to tell whether it helped.
|
||||||
|
- **Changing the OOM behaviour.** The checked-`malloc`-to-catchable-trap path is
|
||||||
|
good and should not be touched; if the measurement finds a hole in it, that is
|
||||||
|
a bug fix, reported separately.
|
||||||
|
- **A memory profiler or allocator instrumentation.** Observability is language
|
||||||
|
iteration 30. RSS from the OS and the existing `time.ticks` are enough for a
|
||||||
|
curve.
|
||||||
|
- **Multi-machine or sharded-across-hosts scaling.** One binary owns its data;
|
||||||
|
cross-process is [9](09-cross-program-tables.md).
|
||||||
|
- **Comparing against SQLite at the ceiling.** `bench/compare/go-sqlite` exists
|
||||||
|
and the comparison would be interesting, but SQLite's whole architecture is
|
||||||
|
the paged design this project rejected — the numbers would not inform any
|
||||||
|
decision here.
|
||||||
|
|
||||||
|
## Info
|
||||||
|
|
||||||
|
Forks the spec must settle:
|
||||||
|
|
||||||
|
1. **What is the limit mechanism for the harness?** A cgroup v2 `memory.max` is
|
||||||
|
the closest thing to how this would actually be deployed; `ulimit -v` is
|
||||||
|
simpler but bounds address space rather than resident set, which for an engine
|
||||||
|
that `malloc`s slabs is a materially different constraint. Leaning cgroup, and
|
||||||
|
the campaign already runs off the fast path so the setup cost is acceptable.
|
||||||
|
2. **Which table shape is the reference?** Per-row overhead depends heavily on
|
||||||
|
whether fields are scalars or heap values — a `Text` column is a separate
|
||||||
|
`db_text` allocation per row, so a text-heavy table and an Int-only table
|
||||||
|
will produce very different slopes. Probably both, reported separately,
|
||||||
|
because "bytes per row" is meaningless without saying which row.
|
||||||
|
3. **Is swap even in scope for the target deployment?** If the intended answer
|
||||||
|
is "run with swap off and let the OOM killer decide", the swap curve is
|
||||||
|
informational rather than load-bearing — but that stance should be stated in
|
||||||
|
the doctrine, not assumed. It also changes which exit iteration 5's
|
||||||
|
back-pressure is defending against.
|
||||||
188
docs/stories/databasev2/02-table-storage-modes.md
Normal file
188
docs/stories/databasev2/02-table-storage-modes.md
Normal file
|
|
@ -0,0 +1,188 @@
|
||||||
|
---
|
||||||
|
track: databasev2
|
||||||
|
iteration: "2"
|
||||||
|
status: refine
|
||||||
|
---
|
||||||
|
|
||||||
|
# databasev2 2 — `@table` storage modes: durability becomes a language decision
|
||||||
|
|
||||||
|
> Part of [Story — databasev2: the database beyond RAM](00-story.md).
|
||||||
|
> Needs [1](01-ram-ceiling-measurement.md) — a mode's default should follow from
|
||||||
|
> a measurement, not a preference.
|
||||||
|
>
|
||||||
|
> **The developer's ask, and the language enrichment this track exists for.**
|
||||||
|
> Today durability is one environment variable for a whole process: `WO_DATA` is
|
||||||
|
> set and every `@table` is WAL-logged, or it is not and none are
|
||||||
|
> (`runtime/src/main.c`). Real applications are not uniform. A session table, a
|
||||||
|
> rate-limit counter and a page cache are resident and disposable; an orders
|
||||||
|
> table is resident and precious; an audit log is precious and rarely read. One
|
||||||
|
> global switch forces "everything is precious" or "nothing is", and the
|
||||||
|
> developer pays for the wrong one either way.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- **Move the storage decision to the declaration site.** `@table(mode: ...)`,
|
||||||
|
chosen per table, in the source, where the person who knows what the data is
|
||||||
|
worth is already writing. An operations runbook is the wrong place for a fact
|
||||||
|
the compiler could hold.
|
||||||
|
- **Three modes, each earning its existence.** `ram` — resident, never logged,
|
||||||
|
gone on restart. `durable` — today's behaviour, resident and ack-after-fsync,
|
||||||
|
and the **default** so every existing program is byte-identical. `cold` —
|
||||||
|
durable and not required to be resident, declared here and *implemented* in
|
||||||
|
[6](06-cold-tiering.md), because a mode with no engine behind it is a promise.
|
||||||
|
- **Make the compiler enforce what the mode means.** This is the part that makes
|
||||||
|
it a language feature rather than a config key. A program that inserts into a
|
||||||
|
`ram` table and expects the row after a restart is stating a contradiction, and
|
||||||
|
the compiler is the right place to say so — at minimum for the cases it can
|
||||||
|
see statically, with the diagnostic catalogued like every other.
|
||||||
|
- **Let the engine act on it.** `ram` tables skip the WAL write entirely, which
|
||||||
|
is not merely a saving — it is the 66× gap iteration 22 measured (4.5k durable
|
||||||
|
vs 297k RAM inserts/s) becoming available per table instead of per process.
|
||||||
|
They also become the first candidates to shed under pressure, which is what
|
||||||
|
[5](05-bounded-tables-eviction.md) builds on.
|
||||||
|
- **Keep principle 7 intact and say why.** RAM stays authoritative for every
|
||||||
|
table that says so. `cold` is a declared, per-table exception a developer opts
|
||||||
|
into with the trade visible at the declaration.
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase A — the grammar
|
||||||
|
|
||||||
|
- Extend the `@table` argument parser with `mode:`. The path is already cut:
|
||||||
|
the argument loop matches `name` and `index` and rejects anything else with a
|
||||||
|
catalogued diagnostic (`unknown @table argument ... (supported: name, index)`),
|
||||||
|
so this is one more arm plus an updated message.
|
||||||
|
- Extend `Ast.table_cfg` — today `{ table_name; indexes }` — with the mode, and
|
||||||
|
give it the default so every existing `@table` keeps its meaning.
|
||||||
|
- Reject the incoherent cases at parse time: `mode` given twice, an unknown mode
|
||||||
|
name. Both belong in the same diagnostic family as the existing `@table`
|
||||||
|
errors, and both go in the error catalog **in this change**, not later — that
|
||||||
|
catalog went stale once by exactly that omission.
|
||||||
|
- Verify: golden AST fixtures for each mode; `woc-test` green; every existing
|
||||||
|
`@table` in every sample parses unchanged.
|
||||||
|
|
||||||
|
### Phase B — the mode reaches the image and the engine
|
||||||
|
|
||||||
|
- Carry the mode through the class descriptor into the `.wob` image so the
|
||||||
|
runtime knows it without re-deriving anything. This is a format change, so it
|
||||||
|
moves `WOB_VERSION` and the format contract in the same commit as the code.
|
||||||
|
- The engine consults it at the one choke point that already exists: the
|
||||||
|
`INDEX HOOK` / WAL staging site in `wo_row_insert` / `wo_row_remove`, which
|
||||||
|
`database/src/CODE-LOGIC.md` names as the only place storage may be mutated.
|
||||||
|
A `ram` table stages nothing.
|
||||||
|
- Replay must skip records for tables that are now `ram` — a WAL written when a
|
||||||
|
table was `durable` and replayed after the source changed is a real
|
||||||
|
migration case, and silently resurrecting rows into a `ram` table would be
|
||||||
|
worse than refusing.
|
||||||
|
- Verify: a `ram` table's inserts produce no WAL growth (measured, not assumed);
|
||||||
|
a `durable` table is byte-identical to today; a mode change across a restart
|
||||||
|
is handled explicitly rather than by accident.
|
||||||
|
|
||||||
|
### Phase C — the compiler's enforcement
|
||||||
|
|
||||||
|
- Decide how far static checking goes (fork 2) and implement that much. The
|
||||||
|
floor: `WO_DATA` set with every table `ram` is a program that asked for a data
|
||||||
|
directory it will never write to — worth a warning at least.
|
||||||
|
- The interesting case is a `ram` table participating in a `ref`/`backlink`
|
||||||
|
relation with a `durable` one. A durable row holding a foreign key into a
|
||||||
|
table that evaporates on restart is a dangling reference by construction, and
|
||||||
|
FK-restrict cannot save it. This is the check most worth having, and it is
|
||||||
|
statically visible from the class table.
|
||||||
|
- Verify: corpus `compile-fail` fixtures for each refusal; each carries the
|
||||||
|
exact `WO-E###` the catalog now documents.
|
||||||
|
|
||||||
|
### Phase D — prove it on a real workload
|
||||||
|
|
||||||
|
- Give `docs/examples/employee` or the `db-bench` sample a mixed schema — at
|
||||||
|
least one `ram` table and one `durable` — and gate the distinction: after a
|
||||||
|
restart, the durable rows are present and the ram rows are gone. That single
|
||||||
|
assertion is the whole feature.
|
||||||
|
- Extend the durability legs of `just db-bench` so the per-table write-path
|
||||||
|
saving appears as a baseline number, not a claim.
|
||||||
|
- Verify: `just employee`, `just db-actor`, `just db-bench`, `oop-accept` green.
|
||||||
|
|
||||||
|
### Phase E — document the contract
|
||||||
|
|
||||||
|
- The `@table` mode surface in the language-surface guide and the db-binding
|
||||||
|
contract; the new diagnostics in the error catalog; the mode's effect on
|
||||||
|
replay in `database/src/CODE-LOGIC.md`.
|
||||||
|
- Verify: `just linkcheck` clean; the language-surface guide's `@table` row
|
||||||
|
matches what the parser actually accepts.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- **Given** every existing `@table` declaration in the repository, **when** it is
|
||||||
|
compiled after this change, **then** behaviour is byte-identical — `durable`
|
||||||
|
is the default and nothing opts in silently.
|
||||||
|
- **Given** a table declared `mode: ram`, **when** rows are inserted with
|
||||||
|
`WO_DATA` set, **then** the WAL does not grow, and after a restart the table is
|
||||||
|
empty while `durable` tables in the same program replay intact.
|
||||||
|
- **Given** `mode: ram` and a measured insert workload, **when** it runs against
|
||||||
|
the same shape as a `durable` table, **then** the write-path saving is visible
|
||||||
|
in `bench/baseline.json` — the per-table half of iteration 22's 66× gap.
|
||||||
|
- **Given** an unknown mode name or `mode:` given twice, **when** it is compiled,
|
||||||
|
**then** it fails with the catalogued diagnostic naming the legal modes.
|
||||||
|
- **Given** a `durable` table holding a `ref` into a `ram` table, **when** it is
|
||||||
|
compiled, **then** the compiler refuses (or warns, per fork 2) — a persistent
|
||||||
|
row cannot reference one that evaporates.
|
||||||
|
- **Given** a WAL containing records for a table whose source now says `ram`,
|
||||||
|
**when** the program starts, **then** the situation is handled explicitly
|
||||||
|
(refuse, or skip and report) and never by silently loading rows into a table
|
||||||
|
declared not to have any.
|
||||||
|
- **Given** the `.wob` format change, **when** an image from the previous version
|
||||||
|
is loaded, **then** the loader refuses it clearly on the version rather than
|
||||||
|
misreading a descriptor.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- **Implementing `cold`.** Declared here so the mode set is settled and the
|
||||||
|
format carries it; the engine behaviour is [6](06-cold-tiering.md). Until then
|
||||||
|
a `cold` declaration must be refused rather than silently treated as
|
||||||
|
`durable` — accepting a mode that does nothing is how a feature becomes a lie.
|
||||||
|
- **Per-table capacity limits and eviction** — [5](05-bounded-tables-eviction.md).
|
||||||
|
This iteration says what a table *is*; that one says how much of it there may
|
||||||
|
be.
|
||||||
|
- **`@table` feature flags and `transaction { }`** — language
|
||||||
|
[iteration 18](../language-runtime-database/18-memory-db-features.md), whose
|
||||||
|
spec is approved and deliberately left whole.
|
||||||
|
- **Per-table WAL files.** One log, one writer, shard 0 — the invariant stage 3
|
||||||
|
established and [7](07-single-file-db.md) depends on. Modes decide *whether* a
|
||||||
|
table logs, never *where*.
|
||||||
|
- **Migrating an existing dataset between modes.** A schema-change story, and
|
||||||
|
the repo already records destructive migrations as a recorded future.
|
||||||
|
- **Encryption at rest, compression of the WAL.** Neither has a consumer.
|
||||||
|
|
||||||
|
## Info
|
||||||
|
|
||||||
|
The grammar surface this touches, read from the source: the parser's `@table`
|
||||||
|
argument loop and its `unknown @table argument` failure; `Ast.table_cfg` as
|
||||||
|
`{ table_name : string option; indexes : string list list }`; and the class
|
||||||
|
descriptor in `runtime/src/wob.h` that the loader validates. Adding a key is
|
||||||
|
genuinely small — the semantics are the iteration.
|
||||||
|
|
||||||
|
Forks the spec must settle:
|
||||||
|
|
||||||
|
1. **What are the modes called?** `ram` / `durable` / `cold` is descriptive of
|
||||||
|
mechanism. `scratch` / `persistent` / `archived` is descriptive of intent and
|
||||||
|
is what a developer reasons about. The names are the API and are hard to
|
||||||
|
change later; leaning the intent-shaped set for the first two if a
|
||||||
|
short-enough pair can be found, since a developer choosing a mode is thinking
|
||||||
|
about what the data is *for*, not about where it sits.
|
||||||
|
2. **How hard does the compiler push?** Three levels: warn on the suspicious
|
||||||
|
cases; refuse the provably-broken ones (a `durable`→`ram` `ref`); or a full
|
||||||
|
dataflow check that an insert into a `ram` table is never expected to persist.
|
||||||
|
The third is not statically decidable in general. Leaning: refuse the
|
||||||
|
relation case (provable, high value), warn on the `WO_DATA`-with-no-durable-
|
||||||
|
table case, and stop there.
|
||||||
|
3. **What is the default, and does it depend on iteration 1?** `durable` keeps
|
||||||
|
every existing program identical, which is nearly decisive. But if iteration
|
||||||
|
1's numbers show the WAL write dominating a workload nobody wanted durable,
|
||||||
|
there is an argument for making the choice mandatory — no default, every
|
||||||
|
`@table` states its mode. That is a bigger source change and a better
|
||||||
|
language; the fork is whether the churn is worth it now or at 1.0.
|
||||||
|
4. **Does `mode: ram` imply anything about the actor/DB-actor path?** Stage 3
|
||||||
|
marshals worker-shard statements to shard 0 because the owner shard holds the
|
||||||
|
store and the WAL. A `ram` table has no WAL — so does it still need to live on
|
||||||
|
the owner? A per-shard `ram` table would be dramatically faster and a
|
||||||
|
different consistency story. Tempting, out of scope here, and worth recording
|
||||||
|
as a candidate rather than deciding in passing.
|
||||||
|
|
@ -1,13 +1,19 @@
|
||||||
---
|
---
|
||||||
iteration: "32"
|
track: databasev2
|
||||||
|
iteration: "3"
|
||||||
|
was_language_iteration: "32"
|
||||||
status: refine
|
status: refine
|
||||||
chain: 6
|
chain: 6
|
||||||
---
|
---
|
||||||
|
|
||||||
# Iteration 32 — WAL checkpoint: disk space reclamation and bounded replay
|
# databasev2 3 — WAL checkpoint: disk space reclamation and bounded replay
|
||||||
|
|
||||||
|
> **Moved 2026-08-26** from the language track, where this was iteration 32.
|
||||||
|
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by
|
||||||
|
> the move; its dependencies are restated in that track index.
|
||||||
|
|
||||||
> Format: `product/story-iteration-template`. Part of
|
> Format: `product/story-iteration-template`. Part of
|
||||||
> [Story — one language, one runtime, one database, one binary](00-story.md).
|
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md).
|
||||||
>
|
>
|
||||||
> **Inserted 2026-08-21** (stage-3 guarantee refinement found the hole):
|
> **Inserted 2026-08-21** (stage-3 guarantee refinement found the hole):
|
||||||
> the WAL is append-only FOREVER — no checkpoint, no truncation exists
|
> the WAL is append-only FOREVER — no checkpoint, no truncation exists
|
||||||
|
|
@ -1,13 +1,19 @@
|
||||||
---
|
---
|
||||||
iteration: "23"
|
track: databasev2
|
||||||
|
iteration: "4"
|
||||||
|
was_language_iteration: "23"
|
||||||
status: refine
|
status: refine
|
||||||
chain: 5
|
chain: 5
|
||||||
---
|
---
|
||||||
|
|
||||||
# Iteration 23 — io_uring group-commit write path
|
# databasev2 4 — io_uring group-commit write path
|
||||||
|
|
||||||
|
> **Moved 2026-08-26** from the language track, where this was iteration 23.
|
||||||
|
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by
|
||||||
|
> the move; its dependencies are restated in that track index.
|
||||||
|
|
||||||
> Format: `product/story-iteration-template`. Part of
|
> Format: `product/story-iteration-template`. Part of
|
||||||
> [Story — one language, one runtime, one database, one binary](00-story.md).
|
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md).
|
||||||
>
|
>
|
||||||
> **Inserted 2026-08-15.** The write-path optimization, and deliberately the
|
> **Inserted 2026-08-15.** The write-path optimization, and deliberately the
|
||||||
> LAST database performance iteration: it only earns its complexity once
|
> LAST database performance iteration: it only earns its complexity once
|
||||||
169
docs/stories/databasev2/05-bounded-tables-eviction.md
Normal file
169
docs/stories/databasev2/05-bounded-tables-eviction.md
Normal file
|
|
@ -0,0 +1,169 @@
|
||||||
|
---
|
||||||
|
track: databasev2
|
||||||
|
iteration: "5"
|
||||||
|
status: refine
|
||||||
|
---
|
||||||
|
|
||||||
|
# databasev2 5 — bounded tables and eviction: a declared budget, and back-pressure before the cliff
|
||||||
|
|
||||||
|
> Part of [Story — databasev2: the database beyond RAM](00-story.md).
|
||||||
|
> Needs [2](02-table-storage-modes.md) for the mode a bound attaches to, and
|
||||||
|
> [1](01-ram-ceiling-measurement.md) for the numbers that set a sane default.
|
||||||
|
>
|
||||||
|
> **The simpler half of the hard problem, done first on purpose.** Evicting from
|
||||||
|
> a bounded resident table and evicting to disk are the same policy question with
|
||||||
|
> different destinations. Getting the policy right where the answer is "drop it"
|
||||||
|
> de-risks [6](06-cold-tiering.md), where the answer is "write it somewhere and
|
||||||
|
> be able to find it again".
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- **A table may declare a maximum.** Rows, bytes, or both — a `ram` table that
|
||||||
|
is a cache or a session store has a size the application is willing to spend,
|
||||||
|
and today it has no way to say so. Unbounded growth in a table nobody intended
|
||||||
|
to be large is the most common route to the ceiling iteration 1 measured.
|
||||||
|
- **Something defined happens at the bound.** Today the answer is "grow until
|
||||||
|
the process dies". The candidates are eviction (drop the least valuable row),
|
||||||
|
refusal (trap, let the caller decide), and back-pressure (make the writer
|
||||||
|
wait). Each is right for a different table, which argues for the policy being
|
||||||
|
declared rather than chosen for the developer.
|
||||||
|
- **Back-pressure before the cliff, not at it.** The dangerous exit iteration 1
|
||||||
|
characterises is swap thrash, which arrives with **no error signal at all**.
|
||||||
|
A budget that is enforced at 100% has already lost; the value is in acting at
|
||||||
|
a threshold, while there is still headroom to act.
|
||||||
|
- **Eviction that respects the engine's actual invariants.** Rows have stable
|
||||||
|
addresses forever, the free-slot list recycles slots, ids are never reused, and
|
||||||
|
every secondary index and unique shadow must stay consistent with the slab.
|
||||||
|
Eviction is `wo_row_remove` with a policy in front — it must go through the
|
||||||
|
same choke point, not around it.
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase A — declaring the bound
|
||||||
|
|
||||||
|
- Extend the `@table` surface from [2](02-table-storage-modes.md) with a
|
||||||
|
capacity and a policy. One new grammar arm, the same catalogued-diagnostic
|
||||||
|
discipline, defaults that keep every existing table unbounded so nothing
|
||||||
|
changes silently.
|
||||||
|
- Decide whether a bound is legal on a `durable` table (fork 1) — evicting a row
|
||||||
|
that was acked as durable is a promise being broken, and the answer is
|
||||||
|
probably "only with an explicit, differently-named policy".
|
||||||
|
- Verify: golden fixtures per policy; existing tables unchanged; illegal
|
||||||
|
combinations refused at compile time with catalogued codes.
|
||||||
|
|
||||||
|
### Phase B — the accounting
|
||||||
|
|
||||||
|
- Track per-table size cheaply. Row count is free; bytes are not — per-row
|
||||||
|
footprint includes the slab slot plus each heap value's own allocation
|
||||||
|
(`db_text`, `db_rec`, `db_multi`, `db_map`), which iteration 1 will have
|
||||||
|
quantified. Decide what is counted and be honest that it is an estimate of RSS,
|
||||||
|
not RSS.
|
||||||
|
- Expose it, because a budget nobody can observe is a budget nobody can tune.
|
||||||
|
How it is exposed is fork 3.
|
||||||
|
- Verify: accounting tracks a known workload within a stated error bound;
|
||||||
|
deleting rows returns the accounting to its prior value (the free-slot list
|
||||||
|
already makes this true of slots).
|
||||||
|
|
||||||
|
### Phase C — the policies
|
||||||
|
|
||||||
|
- **Refuse**: the bound is a hard ceiling, an insert past it traps catchably.
|
||||||
|
The simplest correct behaviour and the right default for anything precious.
|
||||||
|
- **Evict**: drop the least valuable row via `wo_row_remove` so indexes, unique
|
||||||
|
shadows and the free-slot list all stay honest. The recency metadata this needs
|
||||||
|
is fork 2 — and note the engine currently stores no per-row access time, so
|
||||||
|
true LRU is not free.
|
||||||
|
- **Back-pressure**: at a threshold below the bound, slow or park the writer.
|
||||||
|
This composes with the fiber model (a parked writer blocks nobody) and is the
|
||||||
|
only policy that addresses the swap-thrash exit rather than the allocation
|
||||||
|
exit.
|
||||||
|
- Verify: each policy behaves at the bound; after eviction every index agrees
|
||||||
|
with the slab; a parked writer resumes and does not deadlock the shard.
|
||||||
|
|
||||||
|
### Phase D — the pressure signal
|
||||||
|
|
||||||
|
- A process-level threshold, not just per-table: when total resident size
|
||||||
|
crosses a configured fraction, tables with an eviction policy start shedding
|
||||||
|
**before** the allocator or the OS gets involved. This is the iteration's real
|
||||||
|
contribution — turning an invisible failure into a managed one.
|
||||||
|
- Decide precedence when several tables could shed (fork 4).
|
||||||
|
- Verify: under the iteration-1 growth workload with the signal enabled, the
|
||||||
|
process holds a steady state instead of walking into swap; the latency curve
|
||||||
|
stays inside its baseline.
|
||||||
|
|
||||||
|
### Phase E — gate it against the measurement
|
||||||
|
|
||||||
|
- Re-run iteration 1's growth workload with bounds and policies configured. The
|
||||||
|
proof is a before/after on the same harness: previously the curve degraded and
|
||||||
|
the process died; now it plateaus.
|
||||||
|
- Baseline rows for steady-state throughput under pressure.
|
||||||
|
- Verify: `just employee`, `just db-actor`, `just db-bench` green; the gate bites
|
||||||
|
on a doctored pressure metric.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- **Given** a table bounded at N rows with policy `refuse`, **when** the N+1st
|
||||||
|
insert is attempted, **then** it traps catchably, the row count stays N, and
|
||||||
|
every index agrees with the slab.
|
||||||
|
- **Given** a table bounded at N rows with policy `evict`, **when** the N+1st
|
||||||
|
insert arrives, **then** exactly one row is evicted, the new row is present,
|
||||||
|
the count is N, and no index or unique shadow references the evicted row.
|
||||||
|
- **Given** an evicted row's id, **when** it is looked up, **then** it is absent
|
||||||
|
— and its id is never reused by a later insert, preserving the invariant the
|
||||||
|
id hash's tombstone sentinel depends on.
|
||||||
|
- **Given** a bound expressed in bytes, **when** rows of a known shape are
|
||||||
|
inserted, **then** the bound is honoured within the stated accounting error,
|
||||||
|
and that error is documented rather than implied.
|
||||||
|
- **Given** back-pressure configured at a threshold, **when** the threshold is
|
||||||
|
crossed, **then** writers are slowed or parked, reads are unaffected, and no
|
||||||
|
shard deadlocks.
|
||||||
|
- **Given** the process-level pressure signal and the iteration-1 growth
|
||||||
|
workload, **when** it runs to what previously exhausted memory, **then** the
|
||||||
|
process reaches a steady state and read p99 stays within its baseline — the
|
||||||
|
before/after that justifies the iteration.
|
||||||
|
- **Given** a `durable` table, **when** an eviction policy is applied to it,
|
||||||
|
**then** either it is refused at compile time or it is a distinctly named
|
||||||
|
policy that says out loud it discards acked data.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- **Writing evicted rows anywhere** — that is [6](06-cold-tiering.md). Here
|
||||||
|
eviction means the row is gone. Keeping the two apart is what makes the policy
|
||||||
|
work reviewable on its own.
|
||||||
|
- **True LRU if it costs a write per read.** Touching per-row metadata on every
|
||||||
|
read would turn the 1µs read path into a write path — the same trap iteration
|
||||||
|
3's session touch has. An approximation (insertion order, a coarse clock, a
|
||||||
|
sampled counter) is very likely the right answer and fork 2 should say so
|
||||||
|
explicitly rather than defaulting to textbook LRU.
|
||||||
|
- **The TTL cache middleware** — language
|
||||||
|
[iteration 18](../language-runtime-database/18-memory-db-features.md). Expiry
|
||||||
|
by *time* is that; bounding by *size* is this. They compose.
|
||||||
|
- **Query-level result limits.** `take n` already exists in the query surface.
|
||||||
|
- **Shrinking slabs back to the allocator.** Slab addresses are stable forever
|
||||||
|
by design and that invariant is load-bearing; reclaiming a slab whose rows were
|
||||||
|
all evicted is a separate, delicate change with its own iteration if anyone
|
||||||
|
wants it.
|
||||||
|
|
||||||
|
## Info
|
||||||
|
|
||||||
|
Forks the spec must settle:
|
||||||
|
|
||||||
|
1. **May a `durable` table be bounded?** Evicting an acked row contradicts the
|
||||||
|
durability promise. But an audit table that must not grow forever is a real
|
||||||
|
need, and the honest form of it is probably archival (iteration 6) rather than
|
||||||
|
eviction. Leaning: bounds on `durable` are refused, and the need is redirected
|
||||||
|
to 6.
|
||||||
|
2. **What is "least valuable"?** No per-row access time exists today, so LRU
|
||||||
|
costs a write per read. Candidates: insertion order (free — ids are already
|
||||||
|
monotonic per table), a coarse epoch stamped on write only, or sampled
|
||||||
|
approximation. Insertion order is FIFO not LRU, which is wrong for a cache
|
||||||
|
and fine for a queue — so the policy name should say which it is rather than
|
||||||
|
claiming "LRU" and delivering FIFO.
|
||||||
|
3. **How is size observed?** Without observability (language iteration 30) there
|
||||||
|
is no metrics endpoint to publish it on. Options: a builtin returning a
|
||||||
|
table's current size, a `@table`-derived query, or stderr on threshold
|
||||||
|
crossing. A builtin is the smallest thing that makes the feature tunable by
|
||||||
|
the program that owns the budget.
|
||||||
|
4. **Precedence when several tables can shed.** Largest first is simple; the
|
||||||
|
application's own priority order is more correct and needs a way to express
|
||||||
|
it. Proportional shedding is fairest and hardest to reason about. This
|
||||||
|
decides whether the pressure signal is predictable enough to trust.
|
||||||
183
docs/stories/databasev2/06-cold-tiering.md
Normal file
183
docs/stories/databasev2/06-cold-tiering.md
Normal file
|
|
@ -0,0 +1,183 @@
|
||||||
|
---
|
||||||
|
track: databasev2
|
||||||
|
iteration: "6"
|
||||||
|
status: refine
|
||||||
|
---
|
||||||
|
|
||||||
|
# databasev2 6 — cold tiering: rows that leave RAM and come back
|
||||||
|
|
||||||
|
> Part of [Story — databasev2: the database beyond RAM](00-story.md).
|
||||||
|
> Needs [2](02-table-storage-modes.md) for the `cold` mode declaration,
|
||||||
|
> [3](03-wal-checkpoint.md) so the log this builds on does not grow forever,
|
||||||
|
> and [5](05-bounded-tables-eviction.md) for the policy machinery.
|
||||||
|
>
|
||||||
|
> **The iteration that actually raises the ceiling, and the one most likely to
|
||||||
|
> go wrong.** Everything before it makes the limit visible, declared and
|
||||||
|
> managed. This one removes it — for tables that opt in — and in doing so
|
||||||
|
> touches the project's most load-bearing principle. It should be approached
|
||||||
|
> with more suspicion than enthusiasm.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- **A `cold` table may hold more rows than fit in memory.** Recently-used rows
|
||||||
|
are resident; the rest live on disk and are faulted back on access. This is
|
||||||
|
the whole feature and every other goal is a constraint on it.
|
||||||
|
- **Do it without becoming a paged storage engine.** `discarded.md` records that
|
||||||
|
the disk story is the WAL and that a paged B-tree engine was rejected. `cold`
|
||||||
|
must not be a licence to rebuild SQLite inside `database/src`. The design that
|
||||||
|
respects the doctrine reuses the log that already exists plus an index into
|
||||||
|
it — a log-structured read path, not a page cache.
|
||||||
|
- **Keep the resident path exactly as fast as it is.** A `durable` or `ram`
|
||||||
|
table must not pay one instruction for a feature it does not use. Iteration
|
||||||
|
22's 1.3M ops/s read baseline is the regression gate, and a measurable read
|
||||||
|
regression on non-`cold` tables is grounds to reject the design, not to tune
|
||||||
|
it.
|
||||||
|
- **Be honest in the query surface about what a fault costs.** A scan over a
|
||||||
|
`cold` table can touch disk per row. The engine currently answers every read
|
||||||
|
from memory at p50 1µs; a `cold` scan is a different animal and the language
|
||||||
|
should not pretend otherwise — see fork 3, which is the most important
|
||||||
|
question in this iteration.
|
||||||
|
- **Never lose an acked write.** Every guarantee iterations 9 and 22 established
|
||||||
|
holds byte-for-byte: ack-after-fsync, whole-or-nothing replay, torn tails
|
||||||
|
dropped by CRC. A tiering layer that weakens any of those is a regression
|
||||||
|
disguised as a feature.
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase A — settle the design before writing any of it
|
||||||
|
|
||||||
|
- This iteration needs a spec more than any other in the track. The candidate
|
||||||
|
shapes are genuinely different: (a) the WAL becomes the primary store with an
|
||||||
|
in-memory id→offset index and a resident row cache; (b) a separate
|
||||||
|
append-only row file per cold table, checkpointed by 3's machinery; (c)
|
||||||
|
eviction to disk with a free-space map, which is the paged engine wearing a
|
||||||
|
hat.
|
||||||
|
- Whichever wins must state its read amplification, its recovery story, and what
|
||||||
|
happens when the index itself does not fit — an id→offset map for a billion
|
||||||
|
rows is not free either, and a design that only moves the ceiling is worth
|
||||||
|
knowing about before it is built.
|
||||||
|
- Verify: the spec names the shape, the amplification, and the failure modes.
|
||||||
|
No code in this phase.
|
||||||
|
|
||||||
|
### Phase B — the resident/cold boundary
|
||||||
|
|
||||||
|
- Which rows are resident: reuse [5](05-bounded-tables-eviction.md)'s policy and
|
||||||
|
accounting rather than inventing a second notion of "least valuable".
|
||||||
|
- Eviction becomes write-then-drop instead of drop, and it must be atomic with
|
||||||
|
respect to a concurrent reader — a row that is being written out must not be
|
||||||
|
briefly unreachable.
|
||||||
|
- The fault path: a lookup that misses resident memory reads from disk,
|
||||||
|
materialises the row, and admits it under the resident policy.
|
||||||
|
- Verify: a table larger than the resident bound serves correct rows for every
|
||||||
|
id; a row evicted and faulted back is byte-identical, including every heap
|
||||||
|
value (`db_text`, `db_rec`, `db_multi`, `db_map` each round-trip).
|
||||||
|
|
||||||
|
### Phase C — indexes and constraints across the boundary
|
||||||
|
|
||||||
|
- **The hard part, and the reason this is late in the track.** A secondary index
|
||||||
|
over a `cold` table either stays fully resident (bounding the table by index
|
||||||
|
size rather than row size — which may be the honest answer) or is itself
|
||||||
|
tiered. A `@unique` constraint must hold across rows nobody has in memory: the
|
||||||
|
shadow check cannot scan a slab that is not there.
|
||||||
|
- Foreign-key restrict must also hold — a delete has to know whether any cold
|
||||||
|
row references it.
|
||||||
|
- Verify: `@unique` refuses a duplicate whose only conflicting row is cold; FK
|
||||||
|
restrict refuses a delete whose only referrer is cold. These two criteria are
|
||||||
|
the correctness core of the iteration.
|
||||||
|
|
||||||
|
### Phase D — recovery
|
||||||
|
|
||||||
|
- Crash mid-eviction, crash mid-fault, crash mid-checkpoint-of-a-cold-table.
|
||||||
|
Each must recover to a consistent state with no acked write lost and no row
|
||||||
|
visible twice.
|
||||||
|
- Interaction with [3](03-wal-checkpoint.md)'s snapshot: a cold table's on-disk
|
||||||
|
rows are part of the durable state a checkpoint must account for, not
|
||||||
|
something it can truncate past.
|
||||||
|
- Verify: `kill -9` at each of the three points, replayed, with every acked write
|
||||||
|
present and the resident/cold split re-derived correctly.
|
||||||
|
|
||||||
|
### Phase E — measure it, then decide whether to keep it
|
||||||
|
|
||||||
|
- Read/write throughput and p99 for a `cold` table at several resident ratios,
|
||||||
|
and a **regression check that non-`cold` tables did not move**.
|
||||||
|
- Publish the amplification honestly in `perf-targets.md`: how much slower a
|
||||||
|
cold fault is than a resident read, as a number.
|
||||||
|
- Verify: `just db-bench` green with new cold-path rows; the resident baseline
|
||||||
|
unchanged; `just employee`, `just db-actor`, `oop-accept` green; ASan and TSan
|
||||||
|
clean on the fault path.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- **Given** a `cold` table with more rows than the resident bound, **when** any
|
||||||
|
row is looked up by id, **then** it is returned correctly whether resident or
|
||||||
|
faulted, byte-identical including every heap-valued column.
|
||||||
|
- **Given** a `cold` table under a read workload, **when** the resident set is
|
||||||
|
smaller than the working set, **then** the process holds steady state without
|
||||||
|
approaching the RAM ceiling iteration 1 measured.
|
||||||
|
- **Given** a `@unique` column on a `cold` table, **when** a duplicate is
|
||||||
|
inserted whose conflicting row is **not resident**, **then** the insert is
|
||||||
|
refused — the constraint holds across the boundary or it does not hold.
|
||||||
|
- **Given** a `ref` into a `cold` table, **when** the referenced row's owner is
|
||||||
|
deleted and the only referrer is cold, **then** FK restrict refuses the delete.
|
||||||
|
- **Given** `kill -9` during an eviction, a fault, and a checkpoint, **when** the
|
||||||
|
program restarts, **then** every acked write is present, no row appears twice,
|
||||||
|
and the resident/cold split is re-derived correctly.
|
||||||
|
- **Given** a `durable` or `ram` table, **when** the read benchmark runs after
|
||||||
|
this iteration, **then** its throughput and p99 are inside the existing
|
||||||
|
baseline tolerance — no cost for a feature not used.
|
||||||
|
- **Given** a cold fault, **when** its latency is measured, **then** the
|
||||||
|
amplification versus a resident read is recorded in `perf-targets.md` as a
|
||||||
|
number a developer can plan around.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- **A paged B-tree storage engine.** Explicitly rejected in
|
||||||
|
[`discarded.md`](../../plan/discarded.md) and not reopened by this iteration.
|
||||||
|
If the spec phase concludes that tiering *requires* one, the correct outcome is
|
||||||
|
to reject tiering and say so — not to quietly build the thing the project
|
||||||
|
decided against.
|
||||||
|
- **Making `cold` the default, or applying it to a table that did not ask.**
|
||||||
|
Opt-in per table, forever.
|
||||||
|
- **Tiering to anything but the local filesystem.** Object storage needs
|
||||||
|
outbound sockets (language
|
||||||
|
[iteration 38](../language-runtime-database/38-content-platform-capabilities.md))
|
||||||
|
and would change the latency story by orders of magnitude.
|
||||||
|
- **Compression of cold rows.** Composes with
|
||||||
|
[porch 7](../porch/07-sse-and-compression.md)'s codec if that lands first;
|
||||||
|
not a dependency either way and not this slice.
|
||||||
|
- **Cross-shard cold tables.** The owner shard owns the store and the WAL; a
|
||||||
|
cold table is more of the same. Per-shard storage is a separate architectural
|
||||||
|
question noted in [2](02-table-storage-modes.md)'s forks.
|
||||||
|
- **Tiering the query planner's behaviour.** If a scan over a cold table is
|
||||||
|
expensive, the answer for now is that it is expensive and documented — not a
|
||||||
|
cost-based planner.
|
||||||
|
|
||||||
|
## Info
|
||||||
|
|
||||||
|
Forks the spec must settle — this iteration is mostly forks, which is why phase
|
||||||
|
A produces no code:
|
||||||
|
|
||||||
|
1. **Which shape?** WAL-as-primary-store with an id→offset index and a row
|
||||||
|
cache reuses machinery that exists and keeps the doctrine ("the disk story is
|
||||||
|
the WAL") literally true. A separate per-table row file is cleaner to reason
|
||||||
|
about and duplicates the log. Eviction with a free-space map is the rejected
|
||||||
|
paged design. Leaning (a), with the caveat in fork 2.
|
||||||
|
2. **What if the index does not fit either?** An id→offset entry per row is far
|
||||||
|
smaller than a row, so this moves the ceiling by a large constant — but it
|
||||||
|
does not remove it. Say so plainly in the spec: `cold` buys an order of
|
||||||
|
magnitude, not infinity. A design sold as unlimited will be deployed as if it
|
||||||
|
were.
|
||||||
|
3. **Does the language surface the cost?** Three positions. Silent — a cold
|
||||||
|
table reads like any other and the developer discovers the latency in
|
||||||
|
production. Annotated — the mode is at the declaration, so an attentive
|
||||||
|
reader knows, which is the status quo of this design. Or *explicit at the use
|
||||||
|
site*, where a query over a cold table must acknowledge it somehow. The third
|
||||||
|
is most in keeping with a language whose whole thesis is that the compiler
|
||||||
|
tells you the truth — and it is also the most intrusive. This is the fork with
|
||||||
|
the largest effect on what writeonce *is*, and it deserves the brainstorm more
|
||||||
|
than any implementation detail here.
|
||||||
|
4. **Is `@unique` on a cold table simply refused?** Keeping a unique index fully
|
||||||
|
resident is a bound on the table by index size, which is honest and simple.
|
||||||
|
Refusing `@unique` on `cold` outright is even simpler and might be right for
|
||||||
|
a first version — a constraint that silently only checks resident rows would
|
||||||
|
be a correctness hole, and that is the one outcome that must not ship.
|
||||||
|
|
@ -1,12 +1,18 @@
|
||||||
---
|
---
|
||||||
iteration: "33"
|
track: databasev2
|
||||||
|
iteration: "7"
|
||||||
|
was_language_iteration: "33"
|
||||||
status: refine
|
status: refine
|
||||||
---
|
---
|
||||||
|
|
||||||
# Iteration 33 — `WO_DATA=<path>.db`: the persistent store as one file
|
# databasev2 7 — `WO_DATA=<path>.db`: the persistent store as one file
|
||||||
|
|
||||||
|
> **Moved 2026-08-26** from the language track, where this was iteration 33.
|
||||||
|
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by
|
||||||
|
> the move; its dependencies are restated in that track index.
|
||||||
|
|
||||||
> Format: `product/story-iteration-template`. Part of
|
> Format: `product/story-iteration-template`. Part of
|
||||||
> [Story — one language, one runtime, one database, one binary](00-story.md).
|
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md).
|
||||||
>
|
>
|
||||||
> **Inserted 2026-08-22** (developer ask: "can the persistent db be in
|
> **Inserted 2026-08-22** (developer ask: "can the persistent db be in
|
||||||
> file.db form?"). The truth is already almost there: `WO_DATA=<dir>`
|
> file.db form?"). The truth is already almost there: `WO_DATA=<dir>`
|
||||||
|
|
@ -46,7 +52,7 @@ status: refine
|
||||||
|
|
||||||
- A paged database file (SQLite's shape) — RAM is authoritative; the
|
- A paged database file (SQLite's shape) — RAM is authoritative; the
|
||||||
disk story is the WAL, full stop.
|
disk story is the WAL, full stop.
|
||||||
- Checkpoint/compaction — [iteration 32](32-wal-checkpoint.md)'s; its
|
- Checkpoint/compaction — [iteration 32](03-wal-checkpoint.md)'s; its
|
||||||
rename-swap (write snapshot+tail to a NEW file, fsync, `rename()`
|
rename-swap (write snapshot+tail to a NEW file, fsync, `rename()`
|
||||||
over the old) is exactly what keeps the single-file promise crash-safe
|
over the old) is exactly what keeps the single-file promise crash-safe
|
||||||
when it lands. 33 before or after 32 works; landing 33 first means
|
when it lands. 33 before or after 32 works; landing 33 first means
|
||||||
|
|
@ -1,12 +1,18 @@
|
||||||
---
|
---
|
||||||
iteration: "27"
|
track: databasev2
|
||||||
|
iteration: "8"
|
||||||
|
was_language_iteration: "27"
|
||||||
status: hold
|
status: hold
|
||||||
---
|
---
|
||||||
|
|
||||||
# Iteration 27 — query grammar, driven by real embedded-DB corpora
|
# databasev2 8 — query grammar, driven by real embedded-DB corpora
|
||||||
|
|
||||||
|
> **Moved 2026-08-26** from the language track, where this was iteration 27.
|
||||||
|
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by
|
||||||
|
> the move; its dependencies are restated in that track index.
|
||||||
|
|
||||||
> Format: `product/story-iteration-template`. Part of
|
> Format: `product/story-iteration-template`. Part of
|
||||||
> [Story — one language, one runtime, one database, one binary](00-story.md).
|
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md).
|
||||||
>
|
>
|
||||||
> **Inserted 2026-08-16.** A query-surface iteration in the 9b family: the
|
> **Inserted 2026-08-16.** A query-surface iteration in the 9b family: the
|
||||||
> language-integrated query grows to cover the grammar that *real
|
> language-integrated query grows to cover the grammar that *real
|
||||||
|
|
@ -1,12 +1,18 @@
|
||||||
---
|
---
|
||||||
iteration: "20"
|
track: databasev2
|
||||||
|
iteration: "9"
|
||||||
|
was_language_iteration: "20"
|
||||||
status: hold
|
status: hold
|
||||||
---
|
---
|
||||||
|
|
||||||
# Iteration 20 — cross-program tables: attach to a running program's database
|
# databasev2 9 — cross-program tables: attach to a running program's database
|
||||||
|
|
||||||
|
> **Moved 2026-08-26** from the language track, where this was iteration 20.
|
||||||
|
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by
|
||||||
|
> the move; its dependencies are restated in that track index.
|
||||||
|
|
||||||
> Format: `product/story-iteration-template`. Part of
|
> Format: `product/story-iteration-template`. Part of
|
||||||
> [Story — one language, one runtime, one database, one binary](00-story.md).
|
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md).
|
||||||
>
|
>
|
||||||
> **Inserted 2026-08-15**, hence `20`. It follows 9b because a program
|
> **Inserted 2026-08-15**, hence `20`. It follows 9b because a program
|
||||||
> attaching to another's tables wants the same typed statements and queries
|
> attaching to another's tables wants the same typed statements and queries
|
||||||
|
|
@ -141,7 +147,7 @@ read or read+write (per-table refinement deferred until a workload needs
|
||||||
it), and the registration is A's manifest so a grant is a config change +
|
it), and the registration is A's manifest so a grant is a config change +
|
||||||
restart, not an API. **Superseded as the end state (2026-08-15):**
|
restart, not an API. **Superseded as the end state (2026-08-15):**
|
||||||
identity is a keypair and grants name public keys — iteration
|
identity is a keypair and grants name public keys — iteration
|
||||||
[21](21-keypair-attach-auth.md) owns that; the uid check is only this
|
[21](10-keypair-attach-auth.md) owns that; the uid check is only this
|
||||||
iteration's bootstrap and must be flagged pre-21 wherever it ships.
|
iteration's bootstrap and must be flagged pre-21 wherever it ships.
|
||||||
|
|
||||||
**4. What does B's statement actually block on?** B's insert crosses the
|
**4. What does B's statement actually block on?** B's insert crosses the
|
||||||
|
|
@ -1,12 +1,18 @@
|
||||||
---
|
---
|
||||||
iteration: "21"
|
track: databasev2
|
||||||
|
iteration: "10"
|
||||||
|
was_language_iteration: "21"
|
||||||
status: hold
|
status: hold
|
||||||
---
|
---
|
||||||
|
|
||||||
# Iteration 21 — keypair authentication for cross-program attach
|
# databasev2 10 — keypair authentication for cross-program attach
|
||||||
|
|
||||||
|
> **Moved 2026-08-26** from the language track, where this was iteration 21.
|
||||||
|
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by
|
||||||
|
> the move; its dependencies are restated in that track index.
|
||||||
|
|
||||||
> Format: `product/story-iteration-template`. Part of
|
> Format: `product/story-iteration-template`. Part of
|
||||||
> [Story — one language, one runtime, one database, one binary](00-story.md).
|
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md).
|
||||||
>
|
>
|
||||||
> **Inserted 2026-08-15.** Promotes iteration 20's identity fork (Info,
|
> **Inserted 2026-08-15.** Promotes iteration 20's identity fork (Info,
|
||||||
> fork 3) to its own iteration: the name + unix-uid lean is the milestone
|
> fork 3) to its own iteration: the name + unix-uid lean is the milestone
|
||||||
|
|
@ -105,22 +105,17 @@ still pending IS the runtime-concurrency chain; order:
|
||||||
| 16 | 19 | [Float + Bytes](19-missing-scalar-types.md) | **LANDED 2026-08-20** — `.wob` v5; the full stack: IEEE-quiet f64 through literals/VM/@table/WAL/json + Bytes as the binary carrier, no implicit mixing, total-order indexes. Unblocks 24 (WS frames) and the crypto fork (digests). *(was 20)* |
|
| 16 | 19 | [Float + Bytes](19-missing-scalar-types.md) | **LANDED 2026-08-20** — `.wob` v5; the full stack: IEEE-quiet f64 through literals/VM/@table/WAL/json + Bytes as the binary carrier, no implicit mixing, total-order indexes. Unblocks 24 (WS frames) and the crypto fork (digests). *(was 20)* |
|
||||||
| 17 | 31 | [Actor lifecycle](31-actor-lifecycle.md) | request/response (today `send` is one-way and callers `sleep` to await), bounded mailboxes with backpressure (today the FIFO just grows), actor death/supervision, and timers beyond `time.sleep`. 24 cannot be written honestly without these. *(story written 2026-08-21)* |
|
| 17 | 31 | [Actor lifecycle](31-actor-lifecycle.md) | request/response (today `send` is one-way and callers `sleep` to await), bounded mailboxes with backpressure (today the FIFO just grows), actor death/supervision, and timers beyond `time.sleep`. 24 cannot be written honestly without these. *(story written 2026-08-21)* |
|
||||||
| 18 | 24 | [chat: WebSocket workload](24-chat-websocket-workload.md) | the arc's acceptance: WS upgrade + frames (SHA-1 via crypto fork, Bytes via 19), rooms/broadcast, 1k clients, drain-clean. *(was 19)* |
|
| 18 | 24 | [chat: WebSocket workload](24-chat-websocket-workload.md) | the arc's acceptance: WS upgrade + frames (SHA-1 via crypto fork, Bytes via 19), rooms/broadcast, 1k clients, drain-clean. *(was 19)* |
|
||||||
| 19 | 23 | [io_uring group-commit](23-io-uring-commit.md) | WAL WRITE+FSYNC chains on the arc's per-shard rings; fsync fallback kept (after 22 + the arc). *(was 9f)* |
|
|
||||||
| 20 | 32 | [WAL checkpoint](32-wal-checkpoint.md) | **NEW 2026-08-21** (stage-3 guarantee refinement found the hole) — the WAL is append-only forever: snapshot + truncate reclaims disk and bounds replay time; every durability guarantee byte-identical; crash mid-checkpoint recovers from the previous snapshot + full tail. After 23 (composes with group-commit); RAM slot-reuse already contracted in `04-db-binding.md`. |
|
|
||||||
| 21 | 33 | [Single-file store](33-single-file-db.md) | **NEW 2026-08-22** — `WO_DATA=<path>.db`: a file path IS the wal (the store already lives in exactly one file; this makes the surface say so). Driver-only, independent of the chain; composes with 32's rename-swap. |
|
|
||||||
| 22 | 34 | [Crypto builtins](34-crypto-builtins.md) | **NEW 2026-08-22** — SHA-1/SHA-256/HMAC-SHA256 as C builtins over Bytes (no bitwise ops in the language, hand-rolled per doctrine, vector-verified). GATES 24's WS handshake; digest floor for held 21 and the ETag row. |
|
| 22 | 34 | [Crypto builtins](34-crypto-builtins.md) | **NEW 2026-08-22** — SHA-1/SHA-256/HMAC-SHA256 as C builtins over Bytes (no bitwise ops in the language, hand-rolled per doctrine, vector-verified). GATES 24's WS handshake; digest floor for held 21 and the ETag row. |
|
||||||
| 23 | 37 | [wo-html components](37-wo-html-components.md) | **NEW 2026-08-23** — an MVC-shaped view layer in the wo-html LIBRARY (framework stays micro): structural `Component` interface (`render() -> Text`), layout components with slots, the site sample migrated as acceptance. Angular's component FORMAT studied and translated to server-rendered no-JS `.wo`; DI/bindings rejected. **LANDED 2026-08-25.** Raw text literal 2026-08-24 (backtick, verbatim content, margin stripped at lex time, `${ }` raw / `{{ }}` auto-escaping; WO-E004/WO-E005) — lexer plus one parser desugar, nothing downstream. Component layer 2026-08-25: `Component`/`render_all`/`Layout` in wo-html, `ok_html` moved into the framework, site and shop both migrated. |
|
| 23 | 37 | [wo-html components](37-wo-html-components.md) | **NEW 2026-08-23** — an MVC-shaped view layer in the wo-html LIBRARY (framework stays micro): structural `Component` interface (`render() -> Text`), layout components with slots, the site sample migrated as acceptance. Angular's component FORMAT studied and translated to server-rendered no-JS `.wo`; DI/bindings rejected. **LANDED 2026-08-25.** Raw text literal 2026-08-24 (backtick, verbatim content, margin stripped at lex time, `${ }` raw / `{{ }}` auto-escaping; WO-E004/WO-E005) — lexer plus one parser desugar, nothing downstream. Component layer 2026-08-25: `Component`/`render_all`/`Layout` in wo-html, `ok_html` moved into the framework, site and shop both migrated. |
|
||||||
| 23 | 35 | [net runtime seams](35-net-runtime-seams.md) | **NEW 2026-08-22** — the ledger's three 🔧 rows owned: fd deadlines composing with the park plane, Unix-socket listeners, peer address (trusted-proxy check). Framework knobs stay framework slices; pairs naturally with 24 (dead-client eviction). |
|
| 23 | 35 | [net runtime seams](35-net-runtime-seams.md) | **NEW 2026-08-22** — the ledger's three 🔧 rows owned: fd deadlines composing with the park plane, Unix-socket listeners, peer address (trusted-proxy check). Framework knobs stay framework slices; pairs naturally with 24 (dead-client eviction). |
|
||||||
| 23 | 36 | [operator parity](36-operator-parity.md) | **NEW 2026-08-22** — `not`, the five bitwise operators (`& \| ^ << >>`, Int-only, riding the additive/multiplicative rungs Go-style), hex/binary/`_` literals, and compound assigns wired to the written-out form. **Code landed 2026-08-22** on branch `operator-parity`: `.wob` v6, opcodes 42–46 with the 0..63 shift trap (WO-E223), all gates green; reference project `.dev/reference/go` drove the operator-precedence design. Awaiting the developer's MANUAL pass over `docs/examples/operators/` (no test fixtures, by directive). Unblocks story 34's pure-`.wo` HMAC question. |
|
| 23 | 36 | [operator parity](36-operator-parity.md) | **NEW 2026-08-22** — `not`, the five bitwise operators (`& \| ^ << >>`, Int-only, riding the additive/multiplicative rungs Go-style), hex/binary/`_` literals, and compound assigns wired to the written-out form. **Code landed 2026-08-22** on branch `operator-parity`: `.wob` v6, opcodes 42–46 with the 0..63 shift trap (WO-E223), all gates green; reference project `.dev/reference/go` drove the operator-precedence design. Awaiting the developer's MANUAL pass over `docs/examples/operators/` (no test fixtures, by directive). Unblocks story 34's pure-`.wo` HMAC question. |
|
||||||
| 24 | 25 | [HTTP service layer](../../superpowers/plans/2026-08-01-http-service-layer.md) | `service` blocks lower onto the framework (after 9b + 20 by their own precedence notes). **HELD 2026-08-21** — story file removed; the plan doc remains. *(was 10)* |
|
| 24 | 25 | [HTTP service layer](../../superpowers/plans/2026-08-01-http-service-layer.md) | `service` blocks lower onto the framework (after 9b + 20 by their own precedence notes). **HELD 2026-08-21** — story file removed; the plan doc remains. *(was 10)* |
|
||||||
| 25 | 18 | [framework v2: memory-rich features](18-memory-db-features.md) | spec+plan approved: TTL cache, @table flags, durable job queue, `transaction { }` over the WAL's staged batch. **Demoted from seq 14**: more surface on a framework with one consumer, and the cache still stores `Text` because there are no generics |
|
| 25 | 18 | [framework v2: memory-rich features](18-memory-db-features.md) | spec+plan approved: TTL cache, @table flags, durable job queue, `transaction { }` over the WAL's staged batch. **Demoted from seq 14**: more surface on a framework with one consumer, and the cache still stores `Text` because there are no generics |
|
||||||
| 26 | 27 | [Query grammar corpus](27-query-grammar-corpus.md) | grow the query grammar from real corpora; likely collapses to "confirm `len(query)` + add `exists`"; precedes 28. *(was 9g)* |
|
|
||||||
| 27 | 26 | [Blue-green deploy](26-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback (plan authored after 9 + 25). *(was 12)* |
|
| 27 | 26 | [Blue-green deploy](26-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback (plan authored after 9 + 25). *(was 12)* |
|
||||||
| 28 | 20 | [Cross-program tables](20-cross-program-tables.md) | attach to a running program's database over local IPC; owner stays the single writer (channel half-built). **Demoted from seq 16**: new distribution surface while there is no TLS, no crypto, and the multi-shard DB still traps. *(was 9c)* |
|
|
||||||
| 29 | 21 | [Keypair attach auth](21-keypair-attach-auth.md) | program identity is a keypair; mutual challenge–response at attach (crypto half-built; plan folds into 20's). **Demoted with 20** — and it needs crypto primitives that do not exist. *(was 9d)* |
|
|
||||||
| 30 | 28 | [skillhost host workload](28-skillhost-host-workload.md) | host-shaped driving workload naming runtime gaps — demoted with the framework goal. *(was 14)* |
|
| 30 | 28 | [skillhost host workload](28-skillhost-host-workload.md) | host-shaped driving workload naming runtime gaps — demoted with the framework goal. *(was 14)* |
|
||||||
| 31 | 29 | [Compile-time metaprogramming](29-compile-time-metaprogramming.md) | `@derive(...)` from class-table metadata; held with the parked drain by the 2026-08-08 scope directive. *(was 13)* |
|
| 31 | 29 | [Compile-time metaprogramming](29-compile-time-metaprogramming.md) | `@derive(...)` from class-table metadata; held with the parked drain by the 2026-08-08 scope directive. *(was 13)* |
|
||||||
| 32 | 38 | [Content platform capabilities](38-content-platform-capabilities.md) | **NEW 2026-08-26** — the two capability families nothing owns, read off `wob.h`: `fs` mutation (`write`/`remove`/`rename`/`mkdir` — the table holds exactly six fs builtins, ids 40–45, where `append` creates-if-absent and grows, so a file is never replaced, truncated, deleted or renamed) and `net.connect` (ids 51–55 + 91–95, no connect; no `connect()` call in `runtime/src/` at all — which is every identity/notification/federation story at once; 35 deferred connect-side timeouts as "no workload asks yet" — this is the ask). Driven by a `docs/examples/vault` content-collaboration workload in 28's mould; WebDAV, CRDT editing, previews and FTS each get a written verdict instead of an implication. New builtins start at 96 (89/90 are 31's reserved holes); no `.wob` bump. Off-chain, wants a spec. |
|
| 32 | 38 | [Content platform capabilities](38-content-platform-capabilities.md) | **NEW 2026-08-26** — the two capability families nothing owns, read off `wob.h`: `fs` mutation (`write`/`remove`/`rename`/`mkdir` — the table holds exactly six fs builtins, ids 40–45, where `append` creates-if-absent and grows, so a file is never replaced, truncated, deleted or renamed) and `net.connect` (ids 51–55 + 91–95, no connect; no `connect()` call in `runtime/src/` at all — which is every identity/notification/federation story at once; 35 deferred connect-side timeouts as "no workload asks yet" — this is the ask). Driven by a `docs/examples/vault` content-collaboration workload in 28's mould; WebDAV, CRDT editing, previews and FTS each get a written verdict instead of an implication. New builtins start at 96 (89/90 are 31's reserved holes); no `.wob` bump. Off-chain, wants a spec. |
|
||||||
|
| — | — | **[▸ the `databasev2` track](../databasev2/00-story.md)** | **Six pending database iterations moved out 2026-08-26** — WAL checkpoint *(was 32)*, io_uring group commit *(was 23)*, single-file store *(was 33)*, query grammar *(was 27)*, cross-program tables *(was 20)*, keypair attach auth *(was 21)* — renumbered 1–10 in that track alongside four new ones: the RAM-ceiling measurement, `@table` storage modes (the grammar that makes durability per-table instead of one global `WO_DATA`), bounded tables with eviction, and cold tiering. **Done database work stays here as v1 history:** 9 (engine), 9b (`@table`/relations/query) and 22 (durability baseline) are rows above and did not move. |
|
||||||
| 33 | 39 | [Web framework parity](39-web-framework-parity.md) | **NEW 2026-08-26** — gofiber/fiber v3.5.0 added as the web-framework reference and read end to end ([the study](../../plan/exploration/fiber/00-fiber-parity.md)); nine of its 32 middleware already have a `porch` counterpart, so the gaps are breadth, not foundations — with one exception. The study's sharpest finding: the framework's own ledger said CSRF and sessions were UNBLOCKED because iteration 34 landed HMAC, but **there is no source of randomness in the runtime at all**, and an HMAC over a guessable session id is a signed guess. So 39 leads with a random-bytes builtin, then cookies (absent in both directions; `Resp.headers: map<Text,Text>` structurally cannot emit two `Set-Cookie` lines), then the store-backed chain — limiter and idempotency first since they need only `@table` + `time.ticks`. Off-chain, wants a spec. |
|
| 33 | 39 | [Web framework parity](39-web-framework-parity.md) | **NEW 2026-08-26** — gofiber/fiber v3.5.0 added as the web-framework reference and read end to end ([the study](../../plan/exploration/fiber/00-fiber-parity.md)); nine of its 32 middleware already have a `porch` counterpart, so the gaps are breadth, not foundations — with one exception. The study's sharpest finding: the framework's own ledger said CSRF and sessions were UNBLOCKED because iteration 34 landed HMAC, but **there is no source of randomness in the runtime at all**, and an HMAC over a guessable session id is a signed guess. So 39 leads with a random-bytes builtin, then cookies (absent in both directions; `Resp.headers: map<Text,Text>` structurally cannot emit two `Set-Cookie` lines), then the store-backed chain — limiter and idempotency first since they need only `@table` + `time.ticks`. Off-chain, wants a spec. |
|
||||||
| ✅ | 17 | [library projects + `internal/`](17-library-projects-internal.md) | **LANDED 2026-08-20** — `kind = "library"` + entry-less check mode (retires the `--emit` workaround) and Go's `internal/` rule as WO-E108 at the consumer's `use`; driver-only, VM/GC untouched. `just web-app` 26/0 |
|
| ✅ | 17 | [library projects + `internal/`](17-library-projects-internal.md) | **LANDED 2026-08-20** — `kind = "library"` + entry-less check mode (retires the `--emit` workaround) and Go's `internal/` rule as WO-E108 at the consumer's `use`; driver-only, VM/GC untouched. `just web-app` 26/0 |
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -163,7 +163,7 @@ the slice's marker doc when it landed):
|
||||||
| Durability | ✅ fsync-per-commit, ack-after-durable; the ack crosses shards only AFTER the owner's fsync (`just db-actor`'s WAL pair). Power-loss rides fdatasync semantics; 22's kill battery is the scripted proof. |
|
| Durability | ✅ fsync-per-commit, ack-after-durable; the ack crosses shards only AFTER the owner's fsync (`just db-actor`'s WAL pair). Power-loss rides fdatasync semantics; 22's kill battery is the scripted proof. |
|
||||||
| Crash recovery | ✅ boot replay, torn-tail drop, index rebuild; replay completes on the primary before any worker serves (main.c boots the engine before the shards). 22 scripts the restart proof. |
|
| Crash recovery | ✅ boot replay, torn-tail drop, index rebuild; replay completes on the primary before any worker serves (main.c boots the engine before the shards). 22 scripts the restart proof. |
|
||||||
| Concurrency control | ✅ stage 3 — the DB actor serializes every statement; replies are materialized copies, no torn read by construction. Cross-statement snapshots arrive with 18. |
|
| Concurrency control | ✅ stage 3 — the DB actor serializes every statement; replies are materialized copies, no torn read by construction. Cross-statement snapshots arrive with 18. |
|
||||||
| Space reclamation | RAM ✅ (deleted rows free their slot — ids never reused, slots are); disk ✖ → [story 32](32-wal-checkpoint.md), end of chain. |
|
| Space reclamation | RAM ✅ (deleted rows free their slot — ids never reused, slots are); disk ✖ → [databasev2 story 3](../databasev2/03-wal-checkpoint.md), end of chain. |
|
||||||
|
|
||||||
- The C proving ground (`docs/plan/exploration/c-runtime/`, phases A–F:
|
- The C proving ground (`docs/plan/exploration/c-runtime/`, phases A–F:
|
||||||
epoll loops, eventfd mail) is the substrate this lifts into `wovm`.
|
epoll loops, eventfd mail) is the substrate this lifts into `wovm`.
|
||||||
|
|
@ -174,7 +174,7 @@ the slice's marker doc when it landed):
|
||||||
- **Gated by the benchmark:** landing the arc means re-running
|
- **Gated by the benchmark:** landing the arc means re-running
|
||||||
[22](22-durability-throughput-scale.md) at the concurrency
|
[22](22-durability-throughput-scale.md) at the concurrency
|
||||||
scale it unlocks and recording the before/after delta; it is also
|
scale it unlocks and recording the before/after delta; it is also
|
||||||
where [23](23-io-uring-commit.md) gets a thread to overlap
|
where [4](../databasev2/04-io-uring-commit.md) gets a thread to overlap
|
||||||
durability against.
|
durability against.
|
||||||
|
|
||||||
## Proposed Solution
|
## Proposed Solution
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@ Four consumers already wait on it, none able to proceed:
|
||||||
(`Sec-WebSocket-Accept` = base64(SHA-1(key + GUID)) — SHA-1
|
(`Sec-WebSocket-Accept` = base64(SHA-1(key + GUID)) — SHA-1
|
||||||
specifically, not a choice); the framework's ETag/conditional-request
|
specifically, not a choice); the framework's ETag/conditional-request
|
||||||
row (wants a content hash); HMAC-signed tokens the auth core can grow;
|
row (wants a content hash); HMAC-signed tokens the auth core can grow;
|
||||||
and held [iteration 21](21-keypair-attach-auth.md), whose
|
and held [databasev2 10](../databasev2/10-keypair-attach-auth.md), whose
|
||||||
challenge–response needs primitives that "do not exist" (its demotion
|
challenge–response needs primitives that "do not exist" (its demotion
|
||||||
note). Bytes and base64 landed with iteration 19 — the carriers exist,
|
note). Bytes and base64 landed with iteration 19 — the carriers exist,
|
||||||
only the digests are missing.
|
only the digests are missing.
|
||||||
|
|
|
||||||
|
|
@ -93,7 +93,7 @@ status: refine
|
||||||
*does* depend on 28's killable-subprocess work for previews, which is
|
*does* depend on 28's killable-subprocess work for previews, which is
|
||||||
exactly why previews are deferred here rather than attempted.
|
exactly why previews are deferred here rather than attempted.
|
||||||
- **WAL checkpoint and disk reclamation** —
|
- **WAL checkpoint and disk reclamation** —
|
||||||
[iteration 32](32-wal-checkpoint.md). A metadata store whose boot replays
|
[databasev2 3](../databasev2/03-wal-checkpoint.md). A metadata store whose boot replays
|
||||||
every write ever made is a real ceiling for this workload, and naming it
|
every write ever made is a real ceiling for this workload, and naming it
|
||||||
here is the point; fixing it is 32's. This story's gate should record the
|
here is the point; fixing it is 32's. This story's gate should record the
|
||||||
replay time it observes so 32 inherits a number.
|
replay time it observes so 32 inherits a number.
|
||||||
|
|
@ -109,7 +109,7 @@ status: refine
|
||||||
route Nextcloud itself takes is unreachable until `connect` lands.
|
route Nextcloud itself takes is unreachable until `connect` lands.
|
||||||
- **Full-text search.** The engine indexes equality probes on declared
|
- **Full-text search.** The engine indexes equality probes on declared
|
||||||
columns; there is no prefix scan or FTS. Query-grammar growth is
|
columns; there is no prefix scan or FTS. Query-grammar growth is
|
||||||
[iteration 27](27-query-grammar-corpus.md)'s.
|
[databasev2 8](../databasev2/08-query-grammar-corpus.md)'s.
|
||||||
- **TLS** — proxy-terminated, by doctrine, unchanged. The outbound half
|
- **TLS** — proxy-terminated, by doctrine, unchanged. The outbound half
|
||||||
therefore speaks plaintext to a local sidecar or a trusted-network peer,
|
therefore speaks plaintext to a local sidecar or a trusted-network peer,
|
||||||
and the story says so out loud rather than implying HTTPS clients.
|
and the story says so out loud rather than implying HTTPS clients.
|
||||||
|
|
@ -196,9 +196,9 @@ a wish list. Order: brainstorm the five forks (fork 1 and 2 together, fork
|
||||||
establish exactly where it blocks, then the fs verbs, then `connect`, each
|
establish exactly where it blocks, then the fs verbs, then `connect`, each
|
||||||
with corpus fixtures and its own gate leg. Big enough to want a spec and a
|
with corpus fixtures and its own gate leg. Big enough to want a spec and a
|
||||||
plan document — this is not a bounded slice like
|
plan document — this is not a bounded slice like
|
||||||
[33](33-single-file-db.md).
|
[7](../databasev2/07-single-file-db.md).
|
||||||
|
|
||||||
Precedence: independent of the concurrency chain (stage 3 → 22 → 31 → 24 →
|
Precedence: independent of the concurrency chain (stage 3 → 22 → 31 → 24 →
|
||||||
23 → 32) and startable beside it, with one caveat — the honest disk story
|
23 → 32) and startable beside it, with one caveat — the honest disk story
|
||||||
needs [32](32-wal-checkpoint.md), so if this lands first its gate records
|
needs [3](../databasev2/03-wal-checkpoint.md), so if this lands first its gate records
|
||||||
the replay number rather than claiming the platform is operationally done.
|
the replay number rather than claiming the platform is operationally done.
|
||||||
|
|
|
||||||
|
|
@ -115,7 +115,7 @@ status: refine
|
||||||
it.
|
it.
|
||||||
- **Distributed limiting across processes.** One program owns its database;
|
- **Distributed limiting across processes.** One program owns its database;
|
||||||
cross-program state is language
|
cross-program state is language
|
||||||
[iteration 20](../language-runtime-database/20-cross-program-tables.md).
|
[databasev2 9](../databasev2/09-cross-program-tables.md).
|
||||||
- **A background expiry sweeper.** No timer exists (`time.after` is still a
|
- **A background expiry sweeper.** No timer exists (`time.after` is still a
|
||||||
reserved builtin id in `wob.h`). Lazy pruning on access, deliberately.
|
reserved builtin id in `wob.h`). Lazy pruning on access, deliberately.
|
||||||
- **The TTL cache middleware** — language
|
- **The TTL cache middleware** — language
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue