writeonce/docs/superpowers/specs/2026-08-28-wal-group-commit-design.md

# WAL group commit — design

> databasev2 [4](../../stories/databasev2/04-io-uring-commit.md), part A.
> Brainstormed and approved 2026-08-28.
>
> **This spec covers batching only.** The iteration was split during the
> brainstorm: part A amortises one durability barrier across many statements,
> part B (io_uring submission) is deferred until A's measurement says whether
> the blocking boundary is still the bottleneck. That split matches the
> iteration's own fork 1 — "drop-in behind `wo_wal_commit` first, an async
> variant only if the scheduler proves the blocking boundary is the
> bottleneck" — and it means the throughput win arrives behind a much smaller
> correctness surface.

## Decisions taken (the brainstorm's forks, settled)

| Fork | Decision |
| --- | --- |
| Scope | **Batching first, io_uring later.** Two independent wins were being carried as one; only the first needs a new syscall interface, and it is where most of the number lives |
| Batch boundary | **Queue-drain.** Shard 0 stages every pending write request, then commits once. No timer, no tunable |
| Failure | **Fatal, diagnosed abort.** Any failure between "RAM mutated" and "record durable" ends the process |
| Batch cap | **None initially.** Measure peak staged bytes; add a cap only if the queue's existing upstream bound proves insufficient |
| Abort coverage | Unit-test the failure *return*; the abort path itself stays covered by inspection, and that gap is disclosed |

## The problem, read off the engine

The story says "replace fsync-per-commit with io_uring group-commit". Read
against the code, the premise needed correcting: the engine does not commit per
*commit*, it commits per **statement**. `db.c` calls `wo_wal_commit`
immediately after every append, at all six sites — insert, update and remove,
each on both the inline and the DB-actor path. Every single row change is one
`pwrite` plus one `fdatasync`.

That is what the numbers say too. Iteration 22's baseline records durable writes
at **4460 ops/s** single-shard and mixed writes at **1023 ops/s**, p50 **430 µs**,
p99 **664 µs** — against **1.28M ops/s** for durable reads. Writes are roughly
290× slower than reads, and the barrier is the whole of it.

**The batching machinery already exists and is simply never used.**
`wo_wal_commit` writes `w->buf` for `w->len` bytes — a staged buffer that can
hold any number of records. Today it never holds more than one, because the
caller commits immediately after staging. So part A is closer to removing calls
than to adding a mechanism.

## The design

### The commit path

The six `wo_wal_commit` calls come out of `db.c`. Applying to RAM and staging
the record stay exactly where they are; only the barrier moves, up to the point
where shard 0 runs out of work.

Shard 0 owns the WAL — DB statements from other shards arrive as marshaled
request envelopes and are executed on shard 0's thread, serialized, and a reply
envelope unparks the requester. The change is that **the reply is held rather
than sent**: shard 0 executes and stages each queued request, keeps draining
while requests remain, then issues one barrier, and only then releases every
held reply.

Each requester therefore unparks having been acknowledged after the barrier that
carried *its* record — the ack contract the story states, which today is true
only because every batch has one member.

A statement executing inline on shard 0 (rather than arriving as a request)
stages and commits before returning, as it does now. It has no reply to hold —
it returns into its own fiber — and because the drain always commits before it
ends, nothing uncommitted is ever left staged when the inline path runs.

### Why queue-drain, and what it costs

The batch boundary is the queue going empty, not a tick and not a timer. Two
properties follow, and they are the reason to prefer it:

- **A lone writer pays nothing.** One queued request means a batch of one, which
  is today's path at today's latency. Batching engages only under genuine
  contention, so an idle system is not taxed to serve a busy one.
- **The batch self-tunes.** Its size is whatever actually accumulated between
  drains, so it grows with load rather than with a configured number. There is
  nothing to set and nothing to set wrong.

The rejected alternative was the iteration's recorded leaning, the shard tick.
That leaning was recorded when the batch was assumed to ride an io_uring
submission; with batching landing first, a tick boundary would add up to one
quantum of latency even to a lone writer — paying the cost of batching when
there is nothing to batch with.

**No batch cap ships initially, and that is a decision rather than an
oversight.** databasev2 1 established that unbounded growth is precisely how
this engine dies without warning, so the instinct to bound it is right. But the
request queue is already bounded upstream by iteration 24's mailbox caps, and a
second bound on the same quantity is a knob that can only be wrong. The proof
plan measures peak staged bytes so the question is settled by a number.

## Failure: one rule, replacing three behaviours

Today's rollback is uneven, and the code says so. An `insert` whose commit fails
removes the row again, under a comment claiming RAM never claims what disk has
not acknowledged. An `update` or a `delete` whose commit fails does **not** roll
back — its comment admits the state plainly: RAM ahead of disk, trap, do not
ack. Nothing acknowledged is lost, but the process continues with divergent
state, and batching would multiply that from one row to as many as the batch
held.

The rule that replaces it: **once a statement has mutated RAM, the only outcomes
are durable or process death.** It covers both failure points identically —
a staging failure and a barrier failure have the same consequence, RAM ahead of
disk with no way back, and only one of the three verbs can undo itself.

Retrying is not an alternative worth designing for. On Linux a failed `fsync`
may already have discarded the dirty pages, so a second call can report success
having written nothing; the recovery that actually works is replay, which
returns exactly the last durable state. That is what the log is for.

**This removes `WO_T_IO` from the write path.** A program can no longer catch a
disk failure on a write. The removal is deliberate — there was never a
recovery a program could meaningfully perform with its RAM ahead of its disk —
but it is language-visible and must be stated in the story banner and the error
catalogue, not slipped in.

The diagnostic has to earn the abort: the failing operation, the `errno` text,
the WAL path, and the number of records in the batch, on stderr, then exit with
a status of its own. Exit 1 is a trap and exit 2 is a refusal, so a durability
failure takes a third. `abort()` is rejected — a core dump on a full disk is
noise, not evidence.

## What will improve, and what will not

**Corrected 2026-08-28, after reading the baseline properly.** The spec first
pointed at `durable.s1.seed` as the payoff metric. That was wrong, and the
reason is structural rather than a matter of degree.

Worker shards hold no WAL at all — the runtime asserts it — so every DB
statement on a worker marshals to shard 0 and parks, while a statement already
on shard 0 executes inline. **A queue of write requests therefore exists only
when other shards are writing.** Batches form where there is a queue:

| Workload | Today | Batching |
| --- | --- | --- |
| `durable.sN.mixwrite` — concurrent writers across shards | **480 ops/s, p99 5888 µs** | **the target.** N shards marshal N writes and shard 0 pays N barriers serially; one barrier replaces them |
| `durable.s1.mixwrite` — concurrent writers, one shard | 1023 ops/s, p99 664 µs | **no change.** Every write is inline with no queue, so no batch forms |
| `durable.*.seed` — one serial writer | ~4460 ops/s | **no change**, under any batching scheme. There is nothing to batch with |

The inversion in those numbers is the finding worth keeping: **multi-shard
concurrent writes are currently 2× slower than single-shard with a 9× worse
p99.** Adding shards makes durable writing worse today, because every marshaled
statement still buys its own barrier on the owner. That is the pathology group
commit exists to remove, and it is a better argument for this iteration than the
one the story recorded.

**Single-shard concurrent batching is deliberately out of part A.** It would
need the inline path to park its fiber on the barrier rather than commit
synchronously — the same parking machinery part B needs anyway. Deferring it
keeps A to one mechanism, and B inherits the reason to build it.

So the acceptance criterion is scoped: **`durable.sN.mixwrite` throughput up and
its p99 down; `durable.s1.*` and both `seed` legs must not regress.** A plan
that reported "no improvement" against the s1 seed number would be measuring a
workload this change cannot help.

## Proof plan

| Claim | How it is proven |
| --- | --- |
| The payoff is real | **`durable.sN.mixwrite`** before and after on one machine, recorded in `perf-targets.md`. Today 480 ops/s, p99 5888 µs. `durable.s1.*` and both `seed` legs are regression guards, not targets — see the section above |
| Durability is unchanged | Iteration 22's crash battery, unaltered: concurrent writers, `kill -9` mid-stream, replay. **The critical test** — a kill between staging and the barrier must lose only unacknowledged writes |
| Batches actually form | New metrics for mean and peak batch size under contention. If batches are always one, the feature is inert and any throughput change came from somewhere else |
| No idle tax | Single-writer p99 must not regress against the current baseline |
| The cap question is answered | Peak staged bytes recorded per run |
| A failure is detected | `test_wal.c` asserts `wo_wal_commit` reports failure on a bad descriptor |

**One disclosed gap.** Forcing a genuine `fdatasync` failure needs a full or
read-only filesystem, which the gate cannot arrange without mount privileges.
The unit test proves the error is *detected*; the abort that follows it stays
covered by inspection. The alternative — a fault-injection switch — means
shipping a binary that can be told to kill itself, which is a worse trade. This
gap is recorded rather than hidden, because iteration 40 was exactly a fatal
path that nothing exercised.

## Out of scope

- **io_uring submission.** Part B, and it only earns its complexity if A's
  measurement shows the blocking boundary still dominating. A's parking and ack
  machinery is what B would build on, so nothing here is wasted either way.
- **`transaction { }`** — language iteration 18. A transaction already *is* a
  staged batch, so the two compose without either knowing about the other; that
  is a reason not to entangle them now.
- **Checkpoint and compaction** — databasev2 3. This changes when the barrier
  runs, never what the log contains.
- **The read path.** databasev2 1 measured that appending under memory pressure
  costs about 1% while random reads cost 273×, so the pressure is on reads —
  but that is iteration 2's `resident: keys` question, not this one.
- **Rollback with pre-images.** Rejected above: it would add per-write cost on
  every statement to serve a path that ends the process anyway.

## Alternatives rejected

**Tick-boundary batching** — the iteration's recorded leaning, superseded by
the split. It taxes an idle system to serve a busy one.

**Count-or-timer batching** — two tunables, and the timer reintroduces the tick
problem with extra configuration.

**Full rollback with an undo log** — keeps `WO_T_IO` catchable, at the price of
capturing pre-images for every update and delete, paid on every write, to
support continuing in a state the engine cannot trust.

**Keeping today's per-verb behaviour** — turns a rare one-row divergence into a
routine N-row one, silently.