shoney.arickathil da30aa6527 docs: amend principle 7 — the log is authoritative, residency is declared

- driving case: a 120 GB order table on a 32 GB host. Not a tuning problem;
  no eviction policy fixes it. Developer accepted reconsidering the principle
- principle 7 rewritten: durability half UNCHANGED and unconditional
  (WAL-logged, fsync before ack, CRC-dropped torn tail); residency half
  demoted from law to per-table declaration. Old wording quoted in place so
  the amendment is legible, with the reason: a doctrine a real workload
  cannot satisfy gets ignored, and the failure it produced was an OOM kill
- spec: docs/superpowers/specs/2026-08-26-table-residency-design.md
  One log-structured engine — the WAL already holds every row, so keep an
  in-RAM id->offset map and pread rows back. No second engine, no user-space
  row cache (the kernel page cache is the hot copy, which is already this
  repo's stated position and why it avoids O_DIRECT)
- arithmetic that makes it work: 240M rows x 16 B of index = ~3.8 GB
  resident in 32 GB. Indexes stay resident, rows do not. Buys ~2 orders of
  magnitude, not infinity — stated plainly in the spec
- grammar: two optional keys, `durable: true|false` and `resident: all|index`,
  both defaulting to today's behaviour, so all 28 existing @table
  declarations compile untouched and no golden is reblessed
- rejected, with reasons recorded: mmap (rows are pointer-bearing —
  table.c returns (uintptr_t)t as the slot word), buffer pool (the Rust-era
  phase-12 design that died with that track), paged B-tree (stays rejected),
  a three-valued enum, automatic spill, disk-backed-by-default
- self-review caught the budget defaulting to "none" while promising the ERP
  developer a diagnostic instead of the OOM killer — contradiction fixed:
  the budget defaults to a fraction of host memory, and its value comes from
  databasev2 1's swap-onset measurement
- live docs that contradicted the amendment updated (subagent doctrine,
  its guide, discarded.md's two rows, iteration 04's read claim, 07, 38);
  dated specs/plans left as records. linkcheck 0 broken / 0 anchors

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-26 22:42:27 +02:00

3.5 KiB

Raw Blame History

track	iteration	was_language_iteration	status
databasev2	7	33	refine

databasev2 7 — `WO_DATA=<path>.db`: the persistent store as one file

Moved 2026-08-26 from the language track, where this was iteration 33. Part of Story — the database beyond RAM. Content unchanged by the move; its dependencies are restated in that track index.

Format: product/story-iteration-template. Part of Story — one language, one runtime, one database, one binary.

Inserted 2026-08-22 (developer ask: "can the persistent db be in file.db form?"). The truth is already almost there: WO_DATA=<dir> holds exactly ONE file (shard-0.wal) — the entire persistent state, since the log is the whole store and no data pages exist. This iteration makes the surface say so: point WO_DATA at a file and THAT file is the store. Small, driver-only, independent of the concurrency chain.

Goals

WO_DATA=<path> accepts a file path. A path that is not an existing directory is treated as THE wal file (app.db, store.wo.db — the name is the operator's). The directory form stays and keeps meaning <dir>/shard-0.wal — every existing deployment and gate is byte-identical.
One file remains the whole truth at any core count — stage 3 made the WAL owner-shard-only (shard 0 is the sole writer), so nothing multi-shard ever adds a second file.
The contract says it out loud: 04-db-binding.md documents the file form, and documents honestly that the file is an append-only log that grows until iteration 32 lands.

Acceptance Criteria

Given WO_DATA=/tmp/app.db (no such directory), when a program seeds, restarts, and verifies, then replay is byte-true and /tmp/app.db is the only artifact on disk.
Given WO_DATA=<dir> (existing directory), when the same program runs, then behavior is byte-identical to today — <dir>/shard-0.wal, every standing gate unchanged.
Given the db-bench durability legs pointed at the file form, when the restart proof and kill -9 battery run, then every guarantee holds identically (the file IS the same WAL, only named by the operator).

Out Of Scope

A paged database file (SQLite's shape) — still rejected; the disk story is the WAL, full stop.
Checkpoint/compaction — iteration 32's; its rename-swap (write snapshot+tail to a NEW file, fsync, rename() over the old) is exactly what keeps the single-file promise crash-safe when it lands. 33 before or after 32 works; landing 33 first means 32's spec inherits the file form as a stated constraint.
Multiple stores per process, attach-by-file — held iteration 20's territory.

Info

The whole change is runtime/src/main.c's hardcoded snprintf("%s/shard-0.wal", dir) growing a stat-based fork (directory → today's path; otherwise → the path itself), plus a gate check and the binding-doc note. No engine, no WAL format, no compiler.
Fork for the (tiny) spec: what does a NONEXISTENT path mean? Leaning: a path whose parent exists and that does not end in / is a file to create; a trailing / or existing directory keeps the directory form. Refusing ambiguity loudly (WO exit 2) beats guessing.

Proposed Solution

Small enough for a bounded slice: brainstorm the one fork in chat, implement driver + db-bench file-form leg + docs in one pass, gates green. No plan document needed unless it grows.

3.5 KiB Raw Blame History

databasev2 7 — WO_DATA=<path>.db: the persistent store as one file