From b6578ee0a6f4d59a0f875322f883545da895ed48 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Fri, 21 Aug 2026 15:56:13 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20iteration=2022=20spec=20=E2=80=94=20db-?= =?UTF-8?q?bench=20design=20(proposed)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - four story forks settled as leanings; new: db-bench sample vehicle, time.ticks us clock (the one runtime addition) - campaign: ram+durable x single+multi shard, relative gates vs bench/baseline.json + absolute floors, restart proof, kill -9 battery, msgrate (mutex-inbox number), RSS/fd soak discipline Co-Authored-By: Claude Fable 5 --- .../specs/2026-08-21-db-bench-design.md | 171 ++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-21-db-bench-design.md diff --git a/docs/superpowers/specs/2026-08-21-db-bench-design.md b/docs/superpowers/specs/2026-08-21-db-bench-design.md new file mode 100644 index 0000000..004f739 --- /dev/null +++ b/docs/superpowers/specs/2026-08-21-db-bench-design.md @@ -0,0 +1,171 @@ +# Iteration 22 — durability proof, throughput, and scale under load (design) + +**Date:** 2026-08-21 +**Status:** proposed (story iteration 22) — awaiting developer review; the +plan follows approval. Board: [docs/00-status.md](../../stories/00-status.md) +**Scope:** the measurement backbone — a benchmark workload in `.wo`, a +campaign driver script, a tracked baseline contract, and the durability +proofs (restart persistence + crash battery), single- AND multi-shard. +**Relates to:** [story 22](../../stories/language-runtime-database/refine/22-durability-throughput-scale.md) +(the four forks settled below), the landed arc +([story 8's guarantee contract](../../stories/language-runtime-database/done/08-shard-actor-runtime.md) +— the stage-3 delta this iteration records), iteration 23 (the durable +write number it exists to beat), iteration 32 (the aged-store replay +number its policy wants), stage-2 deviation 4 (the mutex-inbox number). + +## Decisions locked during brainstorming (2026-08-21) + +The story's four forks are SETTLED as their recorded leanings (developer +confirmation, the iteration-23 precedent): + +1. **The load generator is compiled `.wo`.** The headline numbers cross + the whole stack — lexer to WAL — because that is the layer a + language-integrated query pays. A C-API microbench exists ONLY to + attribute a regression to engine vs lowering; it is never the quoted + number. +2. **Gates are relative, against a committed baseline.** Fail when a + metric is worse than `bench/baseline.json` by more than its recorded + tolerance (default 15%, tunable per metric in the file). One loud + ABSOLUTE floor per class — a deliberately low catastrophic-regression + tripwire that fails even on a slow machine. The baseline refreshes + only by a commit that says why. +3. **The scale axis is rows, not connections.** "A million users read + and write" means ~1M rows seeded, a bounded-concurrency 90/10 + read/write mix sustained for a stated duration, throughput above the + floor, p99 under the ceiling, RSS flat. Connection scale belongs to + iteration 24's serving workload. +4. **Both durability flavors run and publish, labeled.** `ram` (no + `WO_DATA`) is the engine's ceiling; `durable` (fsync-per-commit) is + the number an operator plans against. The gap between them is + exactly what iteration 23 exists to close — this iteration prices it. + +New decisions (post-arc reality): + +5. **The vehicle is a NEW sample, `docs/examples/db-bench`** — the + employee sample stays a teaching sample; the multi-shard campaign + needs actor writers, which employee is deliberately not. +6. **One runtime addition, disclosed: `time.ticks`** — CLOCK_MONOTONIC + microseconds as an Int. `time.now` is wall-clock milliseconds and + cannot rank microsecond-scale operations; honest per-op p50/p99 from + inside `.wo` needs this clock. It joins the stdlib contract doc like + every builtin; no other runtime change is in scope. + +## 1. Shape — four artifacts + +- **`docs/examples/db-bench`** — the load generator, pure `.wo`, + mode-dispatched on argv exactly as the employee sample is. +- **`scripts/db-bench.sh`** — the campaign driver: environment setup + (`WO_SHARDS`, `WO_DATA`, `WO_IO`), the kill -9 battery, RSS/fd + sampling (the `LW_SOAK` discipline: resident growth beyond 256 KiB or + any descriptor growth fails), result collection into JSON, and the + relative/absolute gate evaluation. +- **`bench/baseline.json`** — the tracked contract: per metric, the + baseline value, its tolerance, and the absolute floor. The first + honest run writes it; every later run is judged against it. +- **`just db-bench`** — runs the campaign, off the fast path (like + `log-watcher::soak`); `just db-bench-quick` runs a seconds-long smoke + of the same modes for CI-shaped sanity, gated loosely. + +## 2. The workload — db-bench's modes + +Two related tables in the employee shape (a parent with a `@unique` +text column, a child with `ref` parent + two indexed columns) so reads, +indexed probes, FK checks, and unique maintenance are all priced. + +Modes, each printing one machine-parsable line per operation class — +` `: + +- **`seed N`** — N child rows (parents amortized), timed inserts. +- **`read N`** — N point-reads by id over the seeded store. +- **`query N`** — N indexed probes (the `where` path). +- **`write N`** — N mixed inserts + field updates. +- **`mix N C`** — the sustained load: 90/10 read/write from C + concurrent actors for N total operations. Single-shard, actors are + fibers on the primary; multi-shard, placement spreads them and every + DB statement rides the stage-3 RPC. +- **`msgrate N`** — the mutex-inbox number: two actors on different + shards ping-pong N messages; reports msgs/sec. This is the number + stage-2's deviation 4 waits on before lock-free rings earn their + complexity (single-shard run included for the same-heap comparison). +- **`verify`** — recount + content checksum + one indexed probe against + expected values; exit nonzero on any mismatch. The restart and crash + proofs are `seed`/`write` runs bracketing a `verify` across process + boundaries. + +Timing is per-operation via `time.ticks`, aggregated in `.wo` +(count, ops/sec, p50, p99 from a fixed-size reservoir — the harness +must stay honest, not clever; if the reservoir ever dominates cost the +spec's answer is a bigger batch, not a fancier sampler). + +## 3. The campaign — what one `just db-bench` run produces + +For each flavor `ram` and `durable`, for each shard count 1 and +default-cores: seed, read, query, write, mix — plus `msgrate` once per +shard count. Results land in one JSON file (per-run, timestamped, in +`bench/results/`, gitignored except the baseline), then the gate +compares against `bench/baseline.json` and fails on any tolerance +breach. The stdout tail is the standup summary: one line per metric +with the baseline delta. + +The FIRST full run on the reference machine writes the baseline and +sets the scale-target numbers (fork 3's floor/ceiling become recorded +values, not prose). The arc's owed before/after is recorded the same +way: the single- vs multi-shard columns of the same table ARE the +delta, stated in the results and copied into story 8's record. + +## 4. Durability — proven by a restart, not asserted + +- **Restart persistence:** `seed` under `WO_DATA`, clean stop, restart, + `verify` — counts, contents, id continuation past the persisted + maximum, and an indexed probe (the index was rebuilt on replay). The + employee gate's existing seed-twice check stays as the second witness. +- **Crash battery:** kill -9 mid-`write` under `WO_DATA`, restart, + `verify` in acknowledged-writes mode: every operation the bench + recorded as acknowledged (it prints a running high-water mark for + exactly this) is present; no partial row is visible. K repetitions + (K recorded in the baseline file), run at BOTH shard counts — the + multi-shard rounds fire the stage-3 obligation under load: a kill + between a worker's send and the owner's commit must leave no ack and + no partial state. +- **Sustained-run hygiene:** during `mix`, the driver samples RSS and + descriptor counts; growth beyond the LW_SOAK tolerances fails the + campaign regardless of throughput. + +## 5. The C-API microbench — attribution only + +A `runtime/test`-shaped harness driving `wo_row_insert`/`wo_row_ptr`/ +probe loops directly, printing the same line format. Run manually when +a headline regression needs blaming (engine vs lowering); never gated, +never quoted. Kept deliberately minimal — one file, no options beyond +counts. + +## 6. Acceptance criteria + +- **Given** a fresh checkout on the reference machine, **when** + `just db-bench` runs to completion, **then** it produces the JSON + results, evaluates every gate against `bench/baseline.json`, and + exits 0 with the summary tail. +- **Given** the restart proof, **when** seed/stop/restart/verify runs + under `WO_DATA`, **then** verify exits 0 (counts, contents, id + continuation, index probe). +- **Given** the crash battery, **when** kill -9 lands mid-write K times + at each shard count, **then** every acknowledged write is present + after replay and no partial state is ever visible — zero tolerance. +- **Given** the multi-shard campaign, **when** the same modes run at + `WO_SHARDS=1` and default cores, **then** both columns publish and + the arc's delta is recorded in story 8; a >tolerance regression in + the single-shard column against baseline fails the gate. +- **Given** a deliberate engine slowdown (a manual smoke, documented in + the plan), **when** the gate runs against the committed baseline, + **then** it FAILS — the contract must be shown to bite before the + iteration closes. + +## Out of scope + +- The optimizations themselves (23's group commit, 32's checkpoint, + ring inboxes) — this iteration prices, they change. +- Cross-host / distributed load; connection-count scale (iteration 24). +- Micro-optimizing the harness; a perfect load generator is not the + deliverable — an honest, repeatable one is. +- A cost-based query planner; group-by (parked with 9b). +- Any runtime change beyond the `time.ticks` builtin.