writeonce/docs/superpowers/plans/2026-08-28-wal-checkpoint.md

databasev2 3 — WAL checkpoint (implementation plan)

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Style rule (user convention): concept, reason, and required behaviour in words plus verification commands only — no implementation or test code blocks; the executor writes the code.

Goal: reclaim disk and bound replay by rewriting the log as one record per live row and swapping it in with rename, so boot replays a short log instead of all history.

Architecture: compaction writes the live store into a temporary file using the existing record grammar and the existing append path, fsyncs it, renames it over the live WAL, fsyncs the parent directory, and reopens the descriptor. Recovery is untouched — boot still opens one file and replays it — and every crash point is safe because rename is atomic.

Tech Stack: C11, libc only. pwrite, fdatasync, rename, open, unlink. No new dependency and no new file format.

Spec: ../specs/2026-08-28-wal-checkpoint-design.md

Global Constraints

• Recovery must not change. No second source, no cutoff offset, no control file. If a task finds itself editing the replay path, something has gone wrong with the design and it should stop rather than proceed.
• Every crash point falls back. Before the rename the live log is untouched; after it the new log is complete. There must be no window in which a reader could observe a mixture.
• The record grammar is frozen. The whole argument for this design is that it already suffices. A compacted log is INSERT records for live rows, ids preserved exactly.
• Bounded memory. stage() grows the staging buffer by doubling and never shrinks it, so dumping a whole store through one buffer would hold the entire store in RAM — the unbounded growth databasev2 1 identified as how this engine dies. The dump must flush periodically.
• Compaction may run only where nothing is staged — in practice immediately after a barrier. Anywhere else, a staged record lands in a file about to be replaced.
• libc only, no new syscall interface. Gates run through just. Never commit on master; branch first.

---

Task 1 — wo_wal_compact: rewrite, fsync, rename, reopen

Files:

• Modify: database/src/wal.c, database/src/wal.h.
• Test: runtime/test/test_wal.c.

Interfaces:

• Produces: a compaction entry point taking the live WAL and the store, which replaces the log with one INSERT record per live row and leaves the WAL usable (descriptor reopened, offset correct). Returns success or failure; a failure must leave the ORIGINAL log intact and usable, because a failed checkpoint is not a durability event.

• Consumes: the existing append path and commit routine, and the bitmap walk that db.c already performs in three places.

• Read three things first and confirm them, because the design rests on them: apply_record implements UPDATE as remove-then-recreate (so records are full row images), wo_wal_append_insert takes an id and reads the row from the store (so ids are preserved), and the tail scan treats a zero length field as end-of-log (so the new file must be zero beyond its records).

• Test first, RED: build a store, age it (insert rows, then update the same rows repeatedly so history exceeds live data), compact, then assert both that the log got materially shorter AND that a fresh replay of it produces the same rows with the same ids and the same values. Shorter alone is worthless — a truncating bug also passes that.

• Verify RED for the right reason: the entry point does not exist yet.

• Implement the walk: for each class, iterate slots via the bitmap and append one INSERT per live row. Reuse the append path; do not write a second encoder.

• Flush every K records rather than staging the whole store. Point a scratch WAL at the temp descriptor and commit periodically. State the chosen K and why in a comment. Without this the dump holds the entire store in RAM.

• Sequence the switch exactly: fsync the temp file, rename over the live path, fsync the parent directory (the rename is atomic in-kernel but the directory entry is not durable until the parent is synced), then reopen the descriptor — the old one refers to an unlinked inode — and reset the offset to the new end of log.

• Handle failure without losing data: any error before the rename must unlink the temp file and leave the live log untouched. A failed compaction is a missed optimisation, not a durability failure, so it must NOT take the fatal path databasev2 4 introduced.

• GREEN: just wovm-test.

• Commit.

Task 2 — a stale temp file is removed, never read

Files:

• Modify: database/src/wal.c (the open path).
• Test: runtime/test/test_wal.c.

Interfaces:

• Consumes: Task 1's temp-file naming.

• Produces: the guarantee that a crash mid-rewrite leaves nothing that can be mistaken for data.

• Test first, RED: place a temp file next to the log containing plausible, well-formed records (not garbage — garbage would be rejected anyway and would prove nothing), open the store, and assert the temp file is gone and the replayed store is exactly what the live log said.

• Verify RED for the right reason.

• Remove any stale temp file when the WAL is opened. Note in a comment why this is safe: the only way one exists is a crash before a rename, and its contents are by definition not yet authoritative.

• GREEN: just wovm-test.

• Commit.

Task 3 — the trigger, and the ordering guard

Files:

• Modify: database/src/wal.c, database/src/wal.h (remember the last compaction's size; the policy decision), runtime/src/vm.c (call the check after the barrier).
• Test: runtime/test/test_wal.c.

Interfaces:

• Consumes: Task 1's compaction entry point.

• Produces: automatic compaction, and the invariant that it never runs with records staged.

• Extract the policy as a pure decision — given the log's used bytes, the bytes the last compaction wrote, and a floor, should we compact? Pure because it is then unit-testable without a store, which is the only way this policy gets tested at all.

• Test the decision directly, RED then GREEN: below the floor it never fires however bad the ratio; above the floor it fires exactly when used bytes exceed the multiple; with no prior compaction it uses the floor alone.

• Record the bytes each compaction wrote, so the denominator is measured rather than estimated. Estimating the live size would mean estimating Text, and the compactor already knows the true number.

• Expose the floor and the ratio as env knobs, matching the existing idiom (WO_MAILBOX, WO_HEAP_MB, WO_SHARDS, WO_WAL_STATS). This is what makes the policy testable — a test sets a tiny floor and forces compaction in a few writes instead of waiting for megabytes. Document them beside the others. Deviation from the spec, disclosed: the spec spoke of a "manual trigger for tests"; env-tunable thresholds serve that purpose without adding language surface, which is the cheaper way to buy the same testability.

• No timer. If the implementer is tempted, the reason is in the spec: Postgres' CheckPointTimeout bounds loss from unflushed buffers, our records are durable at commit, and an idle log does not grow.

• Call the check from the one place that is safe — immediately after the drain's barrier, where nothing is staged. Comment that this is a correctness requirement and not a scheduling preference.

• Verify the guard: a test that stages records and then makes the policy say yes must find compaction deferred, not executed. This is the assertion that keeps the ordering rule true as the code moves.

• Verify durability is unaffected: just db-bench --quick — the crash and restart legs must be unchanged, and part A's wmix legs must still batch.

• Commit.

Task 4 — kill -9 during compaction

Files:

• Test: runtime/test/test_wal.c (extend the existing fork-based crash battery).

Interfaces:

• Consumes: Tasks 1–3.

• Produces: the evidence for the criterion the whole design is shaped around.

• Read the existing crash battery first: a forked child inserts and acks each committed id over a pipe while the parent SIGKILLs it mid-stream, then the parent verifies every acked id survived. Extend that shape rather than inventing a second harness.

• Drive compaction repeatedly in the child (a tiny floor makes it fire often) while it inserts and acks, and kill at many instants so the kill lands inside a rewrite, at the rename, and after it.

• Assert the property, not a state: after replay the store must equal either the pre-compaction or the post-compaction content — never a mixture — and every acked id must be present. A test that only checks "it replayed without error" would pass on a silently truncated log.

• Assert no temp file survives a kill in a way that affects the next boot.

• Run the battery repeatedly, not once: this is a race, and one green run proves very little. State how many repetitions were run in the commit message.

• GREEN: just wovm-test plus the repetitions.

• Commit.

Task 5 — measure: space, boot, and the pause

Files:

• Modify: scripts/db-bench.py (a checkpoint leg), docs/plan/perf-targets.md, bench/baseline.json (refresh, with the reason in the commit message).

Interfaces:

• Consumes: Tasks 1–3.

• Produces: the before/after record, and the pause number the spec deliberately refused to assume.

• Capture the before numbers already measured on master, rather than re-deriving them: seed 20000 leaves a 986 614-byte log; 20 000 updates take it to 2 590 262 bytes with the same live rows; boot+verify on that aged store is 155 ms.

• Add a leg that ages a store, compacts it, and records: bytes before and after, the ratio reclaimed, and boot time before and after. Age it by updating the same rows — history must grow while the live set does not, or the leg is measuring insert throughput instead of compaction.

• Measure the stop-the-world pause on the largest store the harness builds and record it as a number. State the budget it must meet.

• If the pause exceeds the budget, stop and report it. That is the finding the spec asked for, and the alternatives (incremental copy, fork-and-dump) are bought against this number — not before it.

• Give the new metrics tolerances that match what they are: bytes reclaimed is structural and can be gated tightly; the pause is wall-clock on a shared box and cannot. Do not waive them all, which is the mistake part A's task 4 made and had to undo.

• Verify the gate bites: doctor the reclaimed-bytes metric and confirm the suite fails on exactly that metric.

• Refresh the baseline and confirm the full campaign passes against it. The committed baseline is full-mode (N=20000, crash_reps=3) — writing a quick-mode baseline over it is a regression, and part A made exactly that mistake.

• Commit.

Task 6 — closeout

Files:

• Modify: docs/stories/databasev2/03-wal-checkpoint.md, docs/stories/00-status.md, docs/plan/oop-vm/04-db-binding.md, database/src/CODE-LOGIC.md, docs/examples/db-bench/README.md.

• 04-db-binding.md: the normative ordering rule — compaction runs only where nothing is staged, and what recovery does (unchanged: one file, replayed from byte 0). This is the doc the spec named for it.

• CODE-LOGIC.md: why one file rather than snapshot-plus-tail, why rename is the crash-safety primitive, why the dump flushes periodically, and why a failed compaction is not a durability event. Reasoning, not call graph.

• README: the new env knobs beside the existing ones, and the checkpoint leg.

• Story: progress, criteria split met/outstanding, and the measured before/after.

• Board: standup entry in the six-question shape, and the chain note — chain 6 was the last link, so say what the chain's completion means and what is next.

• Record the resident: keys obligation prominently, in the story and at the compactor. Compaction moves every record, so it invalidates every WAL offset iteration 2 stores; the compactor must rebuild that map as it writes. There is nothing to implement today because iteration 2's storage half does not exist — which is exactly why this must be written where the next implementer will hit it, not left in a spec they may not read.

• Full battery: just wovm-test, just woc-test, just oop-e2e, just db-bench, python3 scripts/linkcheck.py .

• Commit.

Self-review notes

• Spec coverage. Compaction and the switch → Task 1. Stale temp → Task 2. Trigger, no timer, ordering rule → Task 3. Crash safety → Task 4. Space, boot, pause → Task 5. Normative doc, resident: keys obligation → Task 6.
• The riskiest task is 4, not 1: Task 1's correctness is a single replay comparison, while Task 4 is a race and can pass by luck. Hence the explicit instruction to run it repeatedly and to state the count.
• Task 2 looks trivial and is not. A stale temp file containing well-formed records is the one input that could be mistaken for data, so the test uses plausible records rather than garbage.
• One thing deliberately NOT a task: rebuilding the resident: keys offset map. It cannot be implemented against a feature that does not exist yet. Recorded as an obligation in Task 6 instead of a stub nobody can test.