writeonce/docs/guides/log-structured-rows.md

Guide — how a resident: keys row lives in the log

For a developer new to this engine. It explains replay, row chains, how a checkpoint flattens them, and why replay of a long chain costs quadratic time. Written 2026-08-30 against the code as it stands, not as a specification — where this disagrees with database/src/wal.c and table.c, they are right.

Everything here concerns @table(resident: keys). A resident: all table — the default — keeps whole rows in memory and none of this applies to it.

---

Replay: how the database gets its memory back

This engine is RAM-authoritative. Tables live in memory and reads never touch disk, which is what makes read latency predictable. Durability is a separate mechanism: every mutation is appended to a write-ahead log before it is acknowledged. Insert a row and two things happen — it goes into an in-memory slab, and an INSERT record goes into the log.

Now kill the process. Memory is gone; the log file is all that survives. So on the next start, before running a line of the program, the runtime opens the log, reads it front to back, and re-applies every record in order — insert, update, delete, delta — rebuilding the tables and their indexes exactly as they were. That is replay: wo_wal_replay_ex looping over records and handing each to apply_record, which dispatches on the record kind.

Three consequences worth internalising:

• The log is the database. RAM is a cache of it that happens to be authoritative while the process lives. That is why an acknowledged write must reach the log before the client is told it succeeded — anything acknowledged but unlogged vanishes on restart.
• Boot time is O(records since the last checkpoint), not O(live data). Insert a million rows and delete them all, with no checkpoint in between, and replay still reads a million inserts and a million tombstones to arrive at an empty table. Checkpointing is what stops that; see below.
• Ordering at boot is delicate. main.c runs replay before wiring rt.wal, so replay lends the runtime a temporary read-only view of the log — a keys-resident row's index maintenance has to read rows back during replay, and there is no live WAL to read through yet. Getting that order wrong made a perfectly good tombstone replay as corruption; that was a real bug.

A resident: keys table has no rows in RAM at all, so without a log there is nothing to reconstruct from. That is why the runtime refuses at startup when such a table is declared and WO_DATA is unset, rather than letting every read return "no such row".

---

The row chain

A row chain is the set of log records that together describe one row's current state, threaded together by back-pointers.

Take a product table — sku, name, price, stock. Insert it, then sell three units in three separate orders:

log (grows rightward, offsets increase) ────────────────────────▶

off 100          off 340        off 372        off 404
┌──────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐
│ INSERT       │ │ DELTA      │ │ DELTA      │ │ DELTA      │
│ sku  "SKU-1" │ │ field=stock│ │ field=stock│ │ field=stock│
│ name "kettle"│ │ value=9    │ │ value=8    │ │ value=7    │
│ price 2999   │ │ back=100   │ │ back=340   │ │ back=372   │
│ stock 10     │ └────────────┘ └────────────┘ └────────────┘
└──────────────┘        ▲              ▲              ▲
       ▲                └──────────────┴──────────────┘
       └──────────────── back-pointers ────────────────┘

id map in RAM:   SKU-1 → 404      ← only the NEWEST offset

Three things to notice:

• A delta is tiny. It carries one field, not the row. Changing stock does not rewrite name or price — which is the whole reason for the design, since a catalogue row is wide and only one narrow field moves per order.
• RAM holds one number per row. Not a list of offsets, just the head of the chain. That is non-negotiable: the id map is the only resident part of a resident: keys table, so if it grew per update the mode would lose its purpose.
• The chain is a backward-linked list living inside the log. Nothing indexes it; the only way to find a delta's predecessor is to read its back-pointer.

Reading a row: the fold

Reading means folding — start at the head, walk backward, take the first value seen for each field, stop at the first full-row record:

404: stock=7   → stock resolved, take 7
372: stock=8   → already resolved, skip
340: stock=9   → already resolved, skip
100: INSERT    → base: fill sku, name, price; stock already resolved
                 result: SKU-1 / kettle / 2999 / 7   ✓

First-seen-wins is what makes newest-wins work, because the walk goes newest-to-oldest. Walk the other direction and the result is stock=9 — plausible, wrong, and invisible unless a test uses two updates to the same field.

wo_wal_fold_row_at is that fold, and there is exactly one of it. Reads, replay and compaction all call it. That is deliberate: a fold that differed between reading and replaying would be a database that changes its mind at boot.

Chain length is the cost variable. A read costs 1 + (chain length) preads. Chain length grows by one per update and resets to zero when a checkpoint rewrites the row.

---

How a checkpoint flattens a chain

Compaction is a rewrite, not an edit, and flattening falls out of that.

wo_wal_compact opens a new temporary log, walks every live row, writes each into it, fsyncs once, then renames the temp over the real log and syncs the parent directory. Garbage in the old log — superseded records, tombstones, whole delta chains — is simply never copied, and disappears when the old inode is unlinked. A crash on either side of that rename leaves either the complete old log or the complete new one, never a half-written one. That is why it is a rename rather than an in-place edit.

Walking live rows, compaction peeks each row's current record kind:

• Not a delta — the bytes are copied verbatim. A row nobody updated is already a single full-row record; re-encoding it would be slower and pointless.
• A delta — stage_flattened_row folds the chain, using the same fold reads use, collecting the newest value per field, and writes one fresh WO_WAL_INSERT containing every field. Not a delta, not the chain: one record.

So the SKU-1 example above becomes a single INSERT carrying SKU-1 / kettle / 2999 / 7. The map is re-pointed at it and the chain is length zero, so the next read costs one pread instead of four.

Details that matter:

• The fold is reused, not reimplemented. If compaction had its own walk, a divergence would silently rewrite rows wrong, permanently, at checkpoint time.
• Compaction refuses to run while records are staged (w->len != 0), so it always sees a fully durable, quiescent log.
• The flattened record's kind is INSERT, not UPDATE. It must replay into an empty database, and there is no earlier record for an UPDATE to modify — that is the point of flattening.

Measured when checkpointing landed: 2.16× space reclaimed, 1.78× faster boot, with a stop-the-world pause of 2 651 µs.

---

Why replay of a long chain is O(N²)

This is the sharp edge, and it is worth understanding as a class of bug rather than one defect.

A delta carries one field. To rebuild a row in RAM — and to fix up its indexes — replay needs the other fields, which live in earlier records. So when replay meets a delta, it folds from that delta's back-pointer to reconstruct the row as it was just before.

Folding at ΔK costs K hops, because the walk must pass every delta between ΔK and the base. Replay meets every delta in the log, and each fold starts over from scratch:

Δ1 → fold walks 1 record
Δ2 → fold walks 2 records
…
ΔN → fold walks N records
        total = 1 + 2 + … + N = N(N+1)/2  →  O(N²)

Each individual fold is linear and looks harmless. The quadratic appears only when you notice replay performs one per delta.

Why not keep the row materialised while replaying it? That would make it linear, and it is blocked by a deliberate choice: keys-resident rows drop their payload after each record precisely so replay's peak memory stays bounded, which is the entire point of the mode. Bounded memory and linear time genuinely conflict here, and the trade has not been revisited.

Why the checkpoint does not save you here

Flattening resets chains to zero, so in principle N stays small. But when compaction runs is decided by WO_CHECKPOINT_BYTES (a floor below which a log is too small to bother with) and WO_CHECKPOINT_RATIO (how much garbage relative to live data triggers a rewrite). Both are ratios over the whole log. Nothing counts per-row chain length.

So one hot row — a single popular SKU taking thousands of small stock updates — barely moves that ratio in a large database. The checkpoint never fires, that row's chain grows without bound, and its replay cost grows as the square of its length. The guard that bounds replay in general is structurally blind to the one case that makes replay quadratic.

This is a known, documented limitation of the shipped feature, not a bug to be surprised by. It is recorded in databasev2 2.

The reviewing lesson

This survived several rounds of review because every fold is correct, every test passes, and the per-call cost is linear and obviously fine. Complexity bugs hide in the caller's loop, not in the function you are reading.

The tell is a linear helper called once per element of the same structure it walks — that shape is quadratic every time. Worth checking whenever a helper starts from a root and a caller invokes it in a loop.