shoney.arickathil
da30aa6527
- driving case: a 120 GB order table on a 32 GB host. Not a tuning problem; no eviction policy fixes it. Developer accepted reconsidering the principle - principle 7 rewritten: durability half UNCHANGED and unconditional (WAL-logged, fsync before ack, CRC-dropped torn tail); residency half demoted from law to per-table declaration. Old wording quoted in place so the amendment is legible, with the reason: a doctrine a real workload cannot satisfy gets ignored, and the failure it produced was an OOM kill - spec: docs/superpowers/specs/2026-08-26-table-residency-design.md One log-structured engine — the WAL already holds every row, so keep an in-RAM id->offset map and pread rows back. No second engine, no user-space row cache (the kernel page cache is the hot copy, which is already this repo's stated position and why it avoids O_DIRECT) - arithmetic that makes it work: 240M rows x 16 B of index = ~3.8 GB resident in 32 GB. Indexes stay resident, rows do not. Buys ~2 orders of magnitude, not infinity — stated plainly in the spec - grammar: two optional keys, `durable: true|false` and `resident: all|index`, both defaulting to today's behaviour, so all 28 existing @table declarations compile untouched and no golden is reblessed - rejected, with reasons recorded: mmap (rows are pointer-bearing — table.c returns (uintptr_t)t as the slot word), buffer pool (the Rust-era phase-12 design that died with that track), paged B-tree (stays rejected), a three-valued enum, automatic spill, disk-backed-by-default - self-review caught the budget defaulting to "none" while promising the ERP developer a diagnostic instead of the OOM killer — contradiction fixed: the budget defaults to a fraction of host memory, and its value comes from databasev2 1's swap-onset measurement - live docs that contradicted the amendment updated (subagent doctrine, its guide, discarded.md's two rows, iteration 04's read claim, 07, 38); dated specs/plans left as records. linkcheck 0 broken / 0 anchors Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
79 lines
3.5 KiB
Markdown
79 lines
3.5 KiB
Markdown
---
|
|
track: databasev2
|
|
iteration: "7"
|
|
was_language_iteration: "33"
|
|
status: refine
|
|
---
|
|
|
|
# databasev2 7 — `WO_DATA=<path>.db`: the persistent store as one file
|
|
|
|
> **Moved 2026-08-26** from the language track, where this was iteration 33.
|
|
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by
|
|
> the move; its dependencies are restated in that track index.
|
|
|
|
> Format: `product/story-iteration-template`. Part of
|
|
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md).
|
|
>
|
|
> **Inserted 2026-08-22** (developer ask: "can the persistent db be in
|
|
> file.db form?"). The truth is already almost there: `WO_DATA=<dir>`
|
|
> holds exactly ONE file (`shard-0.wal`) — the entire persistent state,
|
|
> since the log is the whole store and no data pages exist. This iteration
|
|
> makes the surface say so: point `WO_DATA` at a file and THAT file is
|
|
> the store. Small, driver-only, independent of the concurrency chain.
|
|
|
|
## Goals
|
|
|
|
- **`WO_DATA=<path>` accepts a file path.** A path that is not an
|
|
existing directory is treated as THE wal file (`app.db`,
|
|
`store.wo.db` — the name is the operator's). The directory form stays
|
|
and keeps meaning `<dir>/shard-0.wal` — every existing deployment and
|
|
gate is byte-identical.
|
|
- **One file remains the whole truth at any core count** — stage 3 made
|
|
the WAL owner-shard-only (shard 0 is the sole writer), so nothing
|
|
multi-shard ever adds a second file.
|
|
- **The contract says it out loud**: `04-db-binding.md` documents the
|
|
file form, and documents honestly that the file is an append-only log
|
|
that grows until iteration 32 lands.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- **Given** `WO_DATA=/tmp/app.db` (no such directory), **when** a
|
|
program seeds, restarts, and verifies, **then** replay is byte-true
|
|
and `/tmp/app.db` is the only artifact on disk.
|
|
- **Given** `WO_DATA=<dir>` (existing directory), **when** the same
|
|
program runs, **then** behavior is byte-identical to today —
|
|
`<dir>/shard-0.wal`, every standing gate unchanged.
|
|
- **Given** the db-bench durability legs pointed at the file form,
|
|
**when** the restart proof and kill -9 battery run, **then** every
|
|
guarantee holds identically (the file IS the same WAL, only named by
|
|
the operator).
|
|
|
|
## Out Of Scope
|
|
|
|
- A paged database file (SQLite's shape) — still rejected; the
|
|
disk story is the WAL, full stop.
|
|
- Checkpoint/compaction — [iteration 32](03-wal-checkpoint.md)'s; its
|
|
rename-swap (write snapshot+tail to a NEW file, fsync, `rename()`
|
|
over the old) is exactly what keeps the single-file promise crash-safe
|
|
when it lands. 33 before or after 32 works; landing 33 first means
|
|
32's spec inherits the file form as a stated constraint.
|
|
- Multiple stores per process, attach-by-file — held iteration 20's
|
|
territory.
|
|
|
|
## Info
|
|
|
|
- The whole change is `runtime/src/main.c`'s hardcoded
|
|
`snprintf("%s/shard-0.wal", dir)` growing a stat-based fork
|
|
(directory → today's path; otherwise → the path itself), plus a gate
|
|
check and the binding-doc note. No engine, no WAL format, no
|
|
compiler.
|
|
- Fork for the (tiny) spec: what does a NONEXISTENT path mean? Leaning:
|
|
a path whose parent exists and that does not end in `/` is a file to
|
|
create; a trailing `/` or existing directory keeps the directory
|
|
form. Refusing ambiguity loudly (WO exit 2) beats guessing.
|
|
|
|
## Proposed Solution
|
|
|
|
Small enough for a bounded slice: brainstorm the one fork in chat,
|
|
implement driver + db-bench file-form leg + docs in one pass, gates
|
|
green. No plan document needed unless it grows.
|