shoney.arickathil
026919762b
Plan for the approved spec. Code-free per the repo convention (docs/plan/discarded.md:54); the executor writes the code. - T1 a failed barrier is detected and fatal — one entry point that names the operation, errno, WAL path and batch size, then exits. The abort path itself stays unexercised and the task says so rather than buying coverage with a fault-injection switch - T2 the barrier moves to the drain point and replies are held; the request path stops committing per append. Riskiest task, and its risk is one place: the crash legs. Plan says STOP if they fail, do not adjust the test - T3 the inline path takes the same fatal rule but keeps its own barrier, with a comment explaining the asymmetry so the next reader does not "fix" it. Looks like a no-op; without it the two paths disagree, which is the unevenness the spec exists to remove - T4 prove batches actually form BEFORE measuring the payoff — otherwise a win gets attributed to the wrong cause. Also records peak staged bytes, settling the no-cap decision with a number - T5 measure, gate, write it down. If the payoff is absent, say so and stop: part B must not start on an unproven premise - T6 closeout, including the error catalogue — WO_T_IO leaving the write path is language-visible and must be written down Spec corrected while planning: it pointed at durable.s1.seed as the payoff. Wrong, structurally — worker shards hold no WAL, so a queue only exists when other shards write, and a serial writer has nothing to batch with. The real target is durable.sN.mixwrite: 480 ops/s at p99 5888us against s1's 1023 at p99 664, so adding shards currently makes durable writing WORSE. That inversion is a better argument for the iteration than the one the story recorded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
199 lines
11 KiB
Markdown
199 lines
11 KiB
Markdown
---
|
||
track: databasev2
|
||
iteration: "4"
|
||
was_language_iteration: "23"
|
||
status: in-progress
|
||
chain: 5
|
||
---
|
||
|
||
# databasev2 4 — io_uring group-commit write path
|
||
|
||
> **Moved 2026-08-26** from the language track, where this was iteration 23.
|
||
> Part of [Story — the database beyond RAM](../language-runtime-database/00-story.md). Content unchanged by
|
||
> the move; its dependencies are restated in that track index.
|
||
|
||
> Format: `product/story-iteration-template`. Part of
|
||
> [Story — one language, one runtime, one database, one binary](../language-runtime-database/00-story.md).
|
||
>
|
||
> **Inserted 2026-08-15.** The write-path optimization, and deliberately the
|
||
> LAST database performance iteration: it only earns its complexity once
|
||
> there is a measured fsync-per-commit baseline to beat (iteration 22) and a
|
||
> multithreaded runtime to overlap against (iteration 8). Doing it earlier
|
||
> would optimize a number nobody had measured, against a runtime that
|
||
> couldn't use it.
|
||
>
|
||
> **No spec exists yet.** ~~The forks in *Info* are genuine decisions.~~
|
||
>
|
||
> **REFINED 2026-08-20: the four forks are SETTLED as their recorded
|
||
> leanings** (developer confirmation, no code): (1) drop-in behind
|
||
> `wo_wal_commit` first, an async variant only if the arc's scheduler
|
||
> proves the blocking boundary is the bottleneck; (2) raw
|
||
> `io_uring_setup`/`io_uring_enter` syscalls — libc-only doctrine holds,
|
||
> ring layout documented normatively; (3) the batch boundary is the shard
|
||
> tick (the 8+11 arc's quantum), single-writer fallback batches whatever
|
||
> accumulated; (4) startup auto-probe + an env override so CI proves both
|
||
> paths on one kernel — AMENDED: the override is the arc-wide
|
||
> `WO_IO=uring|epoll` (the arc's T4 owns the probe and the per-shard
|
||
> ring; `WO_WAL_MODE` is subsumed). Position — RE-SEQUENCED 2026-08-21:
|
||
> FIFTH in the concurrency chain (32, WAL checkpoint, follows it —
|
||
> added 2026-08-21), **stage 3 → 22 → 31 → 24 → 23 → 32**
|
||
> (supersedes the 2026-08-20 old-id ordering "9e → 8+11 → 9f"); the
|
||
> per-shard ring already exists (arc T4 landed 2026-08-20,
|
||
> `WO_IO=uring|epoll`) — this iteration adds the WAL's WRITE+FSYNC
|
||
> chains to it. AMENDED 2026-08-20 (io_uring-first
|
||
> directive): the WAL's WRITE+FSYNC chains ride the SAME per-shard ring
|
||
> T4 creates for fiber parking — one event loop per shard, readiness ops
|
||
> and durability ops together, exactly the linux reference project's
|
||
> "single event loop" card. One composition
|
||
> note added since iteration 18: a `transaction { }` already IS a staged
|
||
> batch — under io_uring it becomes exactly one submission, so the two
|
||
> features compose without either knowing the other.
|
||
|
||
> **BRAINSTORMED 2026-08-28 — and SPLIT IN TWO.** Spec for part A:
|
||
> [`2026-08-28-wal-group-commit-design.md`](../../superpowers/specs/2026-08-28-wal-group-commit-design.md)
|
||
> · plan: [`2026-08-28-wal-group-commit.md`](../../superpowers/plans/2026-08-28-wal-group-commit.md)
|
||
> (6 tasks).
|
||
>
|
||
> **The payoff metric is `durable.sN.mixwrite`, not the s1 numbers.** Worker
|
||
> shards hold no WAL — the runtime asserts it — so every statement on a worker
|
||
> marshals to shard 0 and parks, while a statement already on shard 0 runs
|
||
> inline. Batches form only where there is a queue, so concurrent multi-shard
|
||
> writes batch and a single-shard or serial workload does not. The baseline
|
||
> shows why that is the right target anyway: **multi-shard concurrent writes are
|
||
> 480 ops/s at p99 5888 µs against single-shard's 1023 at p99 664 — adding
|
||
> shards makes durable writing WORSE today**, because every marshaled statement
|
||
> still buys its own barrier on the owner.
|
||
>
|
||
> **The premise below needed correcting.** This story says "replace
|
||
> fsync-per-commit with io_uring group-commit", but the engine does not commit
|
||
> per commit — it commits per **statement**: `db.c` calls `wo_wal_commit`
|
||
> immediately after every append, at all six sites, so every row change is one
|
||
> `pwrite` plus one `fdatasync`. That splits the goal into two independent
|
||
> wins, and only the second needs io_uring:
|
||
>
|
||
> - **Part A — batching.** Let many statements share one barrier. The staging
|
||
> buffer already holds any number of records; today it never holds more than
|
||
> one because the caller commits immediately. Mostly a deletion of calls.
|
||
> - **Part B — async submission.** The shard submits and keeps working instead
|
||
> of blocking in `fdatasync`. Deferred until A's measurement says whether the
|
||
> blocking boundary is still the bottleneck.
|
||
>
|
||
> **A is where most of the number lives.** Iteration 22 measured durable writes
|
||
> at 4460 ops/s and mixed writes at 1023 ops/s (p99 664 µs) against 1.28M ops/s
|
||
> for durable reads — ~290× apart, essentially all of it the per-statement
|
||
> barrier.
|
||
>
|
||
> **Forks settled in the brainstorm:** batch boundary is **queue-drain** (not
|
||
> the tick this story recorded — a tick taxes an idle system to serve a busy
|
||
> one); a failure between "RAM mutated" and "record durable" is a **fatal,
|
||
> diagnosed abort**, replacing today's uneven rollback where `insert` undoes
|
||
> itself and `update`/`delete` admit in a comment that they leave RAM ahead of
|
||
> disk. **That removes `WO_T_IO` from the write path** — a language-visible
|
||
> change, recorded here deliberately.
|
||
>
|
||
> `status: in-progress` because the brainstorm is done and the spec is
|
||
> approved; the plan is next. (The `readiness` axis that would say this
|
||
> precisely lives on the unmerged `db-residency-doctrine`.)
|
||
|
||
## Goals
|
||
|
||
- **Replace fsync-per-commit with io_uring group-commit** on the WAL write
|
||
path: batch a tick's committed records into one submission, let the kernel
|
||
overlap the write and the durability barrier, and acknowledge each writer
|
||
only after the barrier its record rode has completed — the same
|
||
ack-after-durable contract, at a fraction of the syscall cost.
|
||
- **Overlap durability with work.** With the shard-actor runtime
|
||
(iteration 8) the shard thread submits its batch and keeps executing ready
|
||
statements while the ring drains, instead of blocking one thread on one
|
||
fdatasync — the multithreading the throughput number has been waiting for.
|
||
- **Keep the durability promise byte-for-byte.** Every guarantee iterations 9
|
||
and 22 proved — replay-whole-or-not-at-all, torn-tail drop, no
|
||
acknowledged write ever lost — holds identically; io_uring changes HOW the
|
||
bytes reach the platter, never WHETHER an ack means durable.
|
||
|
||
## Acceptance Criteria
|
||
|
||
- What to achieve?
|
||
- **Given** the io_uring write path under the iteration-22 crash battery
|
||
(concurrent writers, kill -9 mid-stream, reboot, replay),
|
||
- **when** it runs,
|
||
- **then** every acknowledged write is present after replay and no
|
||
unacknowledged partial write is ever visible — the exact result the
|
||
fsync path gives, so durability is provably unchanged.
|
||
- What to achieve?
|
||
- **Given** the iteration-22 durable write benchmark,
|
||
- **when** it is run on the fsync-per-commit path and then the io_uring
|
||
group-commit path on the same machine,
|
||
- **then** the io_uring path's write throughput is materially higher and
|
||
its p99 commit latency lower, with the before/after numbers recorded —
|
||
the payoff, measured, not asserted.
|
||
- What to achieve?
|
||
- **Given** a kernel without io_uring (old, or restricted by seccomp),
|
||
- **when** the runtime starts,
|
||
- **then** it falls back to the pwrite + fdatasync path automatically and
|
||
correctly — io_uring is an accelerator, never a hard dependency, and a
|
||
binary that runs everywhere is the whole project's premise.
|
||
|
||
## Out Of Scope
|
||
|
||
- **io_uring for the network/accept path.** This iteration is the WAL write
|
||
path only; the socket side is the shard-actor runtime's and the network
|
||
layer's concern.
|
||
- **io_uring for reads.** RAM is authoritative — reads never touch a
|
||
descriptor (phase-B doctrine), so there is nothing to accelerate on the
|
||
read path. This is a write-durability optimization, full stop.
|
||
- **Registered buffers / fixed files / SQPOLL tuning** beyond what the
|
||
benchmark shows is worth it. Start with the plain submit/complete model;
|
||
add ring features only when 22's number says a specific one pays.
|
||
- **Replacing the WAL format or the commit contract.** The bytes on disk and
|
||
the meaning of an ack are iteration 9's; this changes the syscall, not the
|
||
format.
|
||
|
||
## Info
|
||
|
||
Forks the spec must settle:
|
||
|
||
**1. How much of the ring model, and behind what abstraction?** The write
|
||
path today is `pwrite` + `fdatasync` in `database/src/wal.c`; io_uring adds a
|
||
submission/completion queue and a durability barrier op
|
||
(`IORING_OP_FSYNC`/`IORING_FSYNC_DATASYNC` or `O_DSYNC` writes). The fork:
|
||
wrap it behind the existing `wo_wal_commit` boundary (drop-in, the engine
|
||
never learns) or expose an async-commit primitive the shard scheduler drives
|
||
(faster overlap, but couples the WAL to iteration 8's loop). Leaning:
|
||
drop-in behind `wo_wal_commit` first — it is the correctness-preserving
|
||
step and 22 can measure it standalone — then an async variant only if 8's
|
||
scheduler shows the blocking boundary is the remaining bottleneck.
|
||
|
||
**2. liburing or raw syscalls?** liburing is the ergonomic wrapper but is a
|
||
new external dependency, against the libc-only doctrine; the raw
|
||
`io_uring_setup`/`io_uring_enter` syscalls are a few hundred lines and keep
|
||
the doctrine. Leaning: raw syscalls (the doctrine is load-bearing and this is
|
||
a bounded surface), with the mmap'd ring setup written down in the binding
|
||
doc the way the WAL format is — normative, versioned.
|
||
|
||
**3. What is the batch boundary?** Per-statement commit (today) is the
|
||
simplest correct thing and the slowest; a group commit needs a boundary — a
|
||
tick (iteration 8's scheduler quantum), a count, or a short time window.
|
||
Leaning: the shard tick once iteration 8 lands (a batch is "everything
|
||
committed this tick"), with a single-writer fallback that batches whatever
|
||
accumulated between one `wo_wal_commit` call and the ring draining.
|
||
|
||
**4. How is the fallback chosen and tested?** A kernel probe at startup
|
||
(attempt `io_uring_setup`, fall back on ENOSYS/EPERM) is the mechanism; the
|
||
question is how CI proves BOTH paths without two kernels. Leaning: an
|
||
environment override (`WO_WAL_MODE=fsync|uring`) so the test matrix runs the
|
||
crash battery and the benchmark on both on any capable machine, and the
|
||
auto-probe is what production uses.
|
||
|
||
## Proposed Solution
|
||
|
||
- **Brainstorm the spec** after iterations 8 and 22 exist — this iteration is
|
||
meaningless without a multithreaded runtime to overlap against and a
|
||
measured baseline to beat, and its plan's acceptance is literally "22's
|
||
durable number improved, 22's crash battery still green, fsync fallback
|
||
still correct".
|
||
- Expected shape: a `wo_wal` write-mode switch (fsync vs uring), the raw ring
|
||
setup + submit/complete in `database/src/wal.c` (or a `wal_uring.c`
|
||
beside it), the startup probe + `WO_WAL_MODE` override, the binding doc's
|
||
WAL section extended with the ring layout, and iteration 22 re-run on both
|
||
paths with the delta committed.
|