From 3f88b07abbbdc79a01acd7b687d06e13a34d43be Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Sun, 30 Aug 2026 06:34:21 +0200 Subject: [PATCH] docs(db2-delta): implementation plan for keys-resident delta updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 6 tasks: the record kind, the fold, updates + indexes, the request path and group commit, replay + compaction, then lifting the loader refusal and proving it end to end - the fold is written ONCE and called from three places; the tests are arranged to prove each caller separately, and the plan says that wanting a second fold "just for this caller" means the design failed - Task 2 includes a same-field ordering test specifically, because a fold walking the chain backwards the wrong way returns plausible data and is otherwise invisible - Task 6 step 1 audits the db.c request arms BEFORE lifting anything — they were never audited for keys-residency the way the inline path was, and the last audit of that kind found delete corrupting memory - self-review found the spec's crash criterion had no task: added a step that commits a delta, skips the re-point, and replays, which is the state a crash between barrier and flush leaves behind - every symbol the plan names verified to exist in database/src - code-free per house convention; the writing-plans skill wants code blocks and the project rule overrides it Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit abb8fc9454bdca9d8d703c8cfeb224f89a93d14d) --- .../2026-08-30-keys-resident-delta-updates.md | 303 ++++++++++++++++++ 1 file changed, 303 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-30-keys-resident-delta-updates.md diff --git a/docs/superpowers/plans/2026-08-30-keys-resident-delta-updates.md b/docs/superpowers/plans/2026-08-30-keys-resident-delta-updates.md new file mode 100644 index 0000000..0eb4e6d --- /dev/null +++ b/docs/superpowers/plans/2026-08-30-keys-resident-delta-updates.md @@ -0,0 +1,303 @@ +# `resident: keys` delta updates — implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: use +> `superpowers:subagent-driven-development` or `superpowers:executing-plans`, +> task by task. Steps are checkboxes. +> +> **House convention: this plan carries concept, reason and actions in words — +> no implementation or test code blocks.** The engineer writes the code; the +> plan says what must be true and why, and names every file and symbol +> involved. Exact values that must match are quoted inline. + +**Goal:** let a row on a `resident: keys` table be updated, so the loader's +refusal of that annotation can be lifted. + +**Architecture:** an update appends a **delta** record — class, id, field index, +new value, and a back-pointer to the record it supersedes — instead of a full +row. Reading walks the chain backward, first-seen-wins per field, until a +full-row record terminates it. Compaction folds every chain flat, so read cost +is bounded by updates since the last checkpoint rather than by a row's lifetime. + +**Tech stack:** C11, the embedded database under `database/src/` (`wal.c`, +`table.c`, `db.c`), the runtime loader (`runtime/src/loader.c`), the C test +suite (`runtime/test/test_wal.c`), and the `.wo` example under +`docs/examples/residency` gated by `scripts/residency-accept.sh`. + +**Spec:** +[`2026-08-30-keys-resident-delta-updates-design.md`](../specs/2026-08-30-keys-resident-delta-updates-design.md) + +## Global constraints + +- **One fold, three callers.** Reads, replay and compaction must fold + identically. Write it once; no caller may implement its own walk. The spec + names this the design's principal risk: a fold that differs between reading + and replaying is a database that changes its mind at boot. +- **Commit before re-point, always.** A crash after the barrier leaves a + durable delta and a stale map, which replay reconciles. A crash before leaves + the row untouched. Reversing the order breaks both. +- The id map keeps **exactly one slot per row**. A map that grew per update + would defeat the mode's premise, which is that the map is the only resident + part. +- Indexed columns **may** change in a delta. Index maintenance runs per delta. +- Never delete-then-insert as an update. Assigning to a row field writes + through and maintains indexes. +- `resident: all` behaviour and its log records must not change at all. +- Every task ends green on `just wovm-test`; the final tasks also on + `just residency` and `just linkcheck`. +- Commits: bullets only, no prose paragraphs, at most 25 lines, title prefixed + `feat(db2-delta): ` or `fix(db2-delta): `, ending with exactly + `Co-Authored-By: Claude Opus 5 (1M context) `. + Stage explicit paths; never `git add -A`. Do not push, branch or merge. + +## File structure + +| File | Responsibility | +| --- | --- | +| `database/src/wal.h` | the record-kind enum gains a fourth value; the pending-repoint list declaration | +| `database/src/wal.c` | encode/append a delta, **the one fold function**, replay's delta arm, compaction's fold-and-flatten | +| `database/src/table.c` | the update entry points stop refusing and call the append; index maintenance across a delta | +| `database/src/db.c` | the two request-path update arms stage a delta and defer the re-point | +| `runtime/src/loader.c` | the refusal is lifted — the deliverable | +| `runtime/test/test_wal.c` | the C-level proofs, including the resident-table oracle | +| `docs/examples/residency/` + `scripts/residency-accept.sh` | the end-to-end proof that only becomes possible once the refusal lifts | + +The fold lives in `wal.c` because that is where records are read; `table.c` +calls it through the existing `wo_row_borrow`. + +--- + +### Task 1 — the record kind and its encoding + +**Files:** modify `database/src/wal.h` (the enum at line 46, +`WO_WAL_INSERT = 1, WO_WAL_REMOVE = 2, WO_WAL_UPDATE = 3`) and +`database/src/wal.c`. + +**Produces:** `WO_WAL_DELTA = 4`, and an internal append that stages one delta +record. A delta's payload is: kind byte, class id, row id, field index, the +back-pointer offset, then the field's value encoded by the existing `enc_val`. + +- [ ] **Step 1 — extend the enum.** Add the fourth value. Do not renumber the + existing three: they are on disk in every log that exists. +- [ ] **Step 2 — write the failing test.** In `runtime/test/test_wal.c`, stage a + delta, commit, then read the raw record back with `scan_record` and assert + every field round-trips: kind, class, id, field index, back-pointer, and + the value. It must fail before the encoder exists. +- [ ] **Step 3 — run it and watch it fail.** `make -C runtime build/test_wal && + ./runtime/build/test_wal`. Expect a compile failure naming the missing + symbol — that is a legitimate first failure for C. +- [ ] **Step 4 — implement the append.** Follow `wo_wal_append_insert`'s shape: + build a `wbuf`, `wput_*` the header fields, `enc_val` the one value, and + `stage()` it. Take the back-pointer as a parameter rather than reading it + from the map — the caller knows it and this keeps the encoder ignorant of + table state. +- [ ] **Step 5 — run the test.** Expect pass. +- [ ] **Step 6 — commit.** Prefix `feat(db2-delta)`. + +--- + +### Task 2 — the fold, and reads through it + +**Files:** modify `database/src/wal.c` (near `wo_wal_read_row_at`, line ~600) +and `database/src/table.c` (`wo_row_borrow`). + +**Consumes:** `WO_WAL_DELTA` and the record layout from Task 1. + +**Produces:** the single fold function every later task calls. Given a starting +offset it walks back through delta records, remembering the first value seen for +each field index, until it meets an INSERT or UPDATE record; it materialises +that base row and overlays the remembered fields. Name it once and do not +duplicate it. + +- [ ] **Step 1 — write the failing test.** Insert a keys-resident row, drop its + payload, append two deltas to two *different* fields, re-point after each, + then borrow the row and assert both changed fields carry their new values + and the untouched fields carry their originals. +- [ ] **Step 2 — write a second failing test that pins the ordering rule.** Two + deltas to the SAME field. The newer must win. This is the test that + catches a fold walking the chain in the wrong direction, which is + otherwise invisible — a wrong-direction fold returns plausible data. +- [ ] **Step 3 — run both, watch them fail.** +- [ ] **Step 4 — implement the fold.** Walk backward from the given offset. + Track which field indexes have been resolved; skip a delta for a field + already resolved, because the newer one was seen first. Stop at the first + full-row record and decode it, then overlay. Guard against a cycle in the + back-pointers by refusing to walk more records than the log can hold — + corrupt back-pointers must fail loudly, not spin. +- [ ] **Step 5 — route `wo_row_borrow` through it.** Its keys arm currently + calls `wo_wal_read_row_at` directly; it now folds instead. Everything the + borrow already guarantees stays: the per-table scratch, the `scratch_busy` + nested-borrow refusal, and the cid/id identity check. +- [ ] **Step 6 — run the whole WAL suite.** Every existing keys test must still + pass — a row with no deltas folds to itself, which is the case they cover. +- [ ] **Step 7 — commit.** Prefix `feat(db2-delta)`. + +--- + +### Task 3 — updates append, indexes follow + +**Files:** modify `database/src/table.c` (`wo_row_update_field` and +`wo_row_update_field_slot` — both currently refuse keys tables with +"update on a `resident: keys` table is not implemented") and +`database/src/wal.c`. + +**Consumes:** the append from Task 1, the fold from Task 2. + +**Produces:** a working update on a keys-resident table, and the pending-repoint +list the request path needs in Task 4. + +- [ ] **Step 1 — write the failing test.** Update one field of a keys-resident + row through `wo_row_update_field`, then read it back and assert the new + value. It fails today with the refusal message. +- [ ] **Step 2 — write the index test, which is the one that matters.** Give the + table a secondary index, update an **indexed** column, then query through + that index and assert the row is found by its NEW value and NOT by its + old. The spec allows indexed columns to change specifically because a + catalogue indexes `price`, so this is the realistic case, not an edge one. +- [ ] **Step 3 — run both, watch them fail.** +- [ ] **Step 4 — replace the refusals.** Both entry points converge on one + internal path: borrow the row (which folds), read its current offset with + `wo_row_offset1`, append the delta with that offset as the back-pointer, + commit, then `wo_row_set_offset` to the new record. +- [ ] **Step 5 — maintain the indexes.** The borrowed row is the OLD row; call + `idx_remove_row` with it, apply the field change to the materialised copy, + then `idx_add_row`. Both functions already exist in `table.c` (lines 452 + and 487) and both take a `db_row *`. +- [ ] **Step 6 — release the borrow on every exit**, including the failure + arms. A keys borrow holds the per-table scratch and blocks the next + borrow until released; an early return that skips it deadlocks the table. +- [ ] **Step 7 — run the suite.** Expect pass. +- [ ] **Step 8 — commit.** Prefix `feat(db2-delta)`. + +--- + +### Task 4 — the request path and group commit + +**Files:** modify `database/src/db.c` (the two `WO_B_DB_UPDATE_FIELD` arms at +lines 88 and 289 — one inline, one on the request path) and +`database/src/wal.c` (the pending list beside `pend`/`pend_len`/`pend_cap` in +`wal.h`, and `wo_db_flush_drops`). + +**Consumes:** the update path from Task 3. + +- [ ] **Step 1 — read how inserts already do this before writing anything.** + The insert path stages, lets the drain commit once at a barrier, and only + then applies its map change through `wo_db_flush_drops`. An update must + follow the same shape: the re-point is a map change that must not happen + before the barrier. +- [ ] **Step 2 — write the failing test.** Two updates to the same row inside + one drain, then a read. Both must be visible, and the row's chain must + have both deltas. A re-point applied too early shows up as the second + delta's back-pointer skipping the first. +- [ ] **Step 3 — run it, watch it fail.** +- [ ] **Step 4 — add the pending-repoint list.** Mirror the pending-drop list's + declaration and lifetime exactly. Do not reuse the drop list: a drop and a + re-point carry the same three fields but mean different things, and + merging them would make a future reader guess. +- [ ] **Step 5 — wire both arms.** The inline arm commits and flushes + immediately, as it does for inserts; the request arm records the pending + re-point and lets the drain's barrier flush it. +- [ ] **Step 6 — run the suite, and `just db-actor`** — that gate drives actors + on worker shards through the request path, which is the arm most easily + got wrong. +- [ ] **Step 7 — commit.** Prefix `feat(db2-delta)`. + +--- + +### Task 5 — replay and compaction fold the same way + +**Files:** modify `database/src/wal.c` (`apply_record` at line 738, +`wo_wal_replay_ex` at line ~861, and `wo_wal_compact`). + +**Consumes:** the fold from Task 2. + +- [ ] **Step 1 — write the failing replay test.** Build a row with a chain of + three deltas, close the log, replay it into a fresh database, and assert + the row reads exactly as it did before the restart. Assert the indexes too: + query through a secondary index whose column one of the deltas changed. +- [ ] **Step 2 — write the failing compaction test.** Same chain, then compact, + then assert the row still reads identically **and its chain is now length + zero** — the record its offset points at must be a full row, not a delta. + That second assertion is the one proving compaction is the bound the spec + relies on; without it the test passes even if compaction copied the chain. +- [ ] **Step 3 — run both, watch them fail.** +- [ ] **Step 4 — replay's delta arm.** A delta re-points the row's offset, the + same way a dropped payload does. Because indexed columns may change, + replay must also fold the delta into the indexes built from the base + record. Note `wo_wal_replay_ex` already lends the runtime a read-only WAL + view for the replay's duration — the fold needs it and it is already there. +- [ ] **Step 5 — compaction folds flat.** It already walks live ids and copies + each row's current record. It must now fold each row and write ONE + full-row record, so every checkpoint resets every chain. Keep the existing + byte-for-byte copy for rows whose chain is already empty: it is faster and + it is what the current code does. +- [ ] **Step 6 — the crash window, which is the criterion nothing else covers.** + Append a delta, commit it, and then do NOT re-point the map — that is + exactly the state a crash between the barrier and the flush leaves behind. + Replay the log into a fresh database and assert the update IS present. + This is the test proving the commit-before-re-point ordering earns its + place; without it the ordering is an untested assertion in a comment. + `test_compact_crash_battery` in the same file shows the established shape + for simulating a crash point without actually killing a process. +- [ ] **Step 7 — run the suite plus `just residency`.** +- [ ] **Step 8 — commit.** Prefix `feat(db2-delta)`. + +--- + +### Task 6 — lift the refusal, and prove it end to end + +**Files:** modify `runtime/src/loader.c` (the `WO_CLASSF_RESIDENT_KEYS` BAIL), +`docs/examples/residency/main.wo`, `docs/examples/residency/README.md`, +`scripts/residency-accept.sh`, `docs/stories/databasev2/02-table-storage-modes.md`, +`docs/stories/00-status.md`. + +**Consumes:** everything above. + +- [ ] **Step 1 — audit the request path before lifting anything.** The `db.c` + request arms were never audited for keys-residency the way the inline + path was, and the last audit of that kind found `delete` corrupting + memory. Read every arm that touches a row on a keys table and confirm it + goes through borrow/release or `wo_row_next_id`, never `wo_row_ptr` + directly. Report what you checked, not just the conclusion. +- [ ] **Step 2 — lift the loader refusal.** Delete the BAIL. Keep the second + one: `durable: false` with `resident: keys` stays refused, because a row + that is never logged has nowhere to be read from. +- [ ] **Step 3 — make the example real.** In `docs/examples/residency/main.wo`, + uncomment the keys-resident `Product` declaration and make it the live + one. `place_order` already updates `stock`, which is now the feature under + test rather than the thing that cannot work. +- [ ] **Step 4 — gate it.** `scripts/residency-accept.sh` currently asserts the + annotation is REFUSED at load. That leg must invert: it now asserts the + program runs and the order decrements stock across a restart. Delete the + refusal assertion rather than leaving it inverted-but-present. +- [ ] **Step 5 — the oracle test.** In `test_wal.c`, run the same sequence of + updates against a `resident: all` table and a `resident: keys` table and + assert the rows are identical at every step. The resident table is the + oracle; this is the strongest available check that the fold agrees with + ordinary storage. +- [ ] **Step 6 — update the docs to say what is now true.** The story's + Outstanding criterion for updates moves to Met, recording HOW it was + verified. The example's README loses its "why it is refused" section and + gains what the mode costs: a read is one pread plus deltas since the last + checkpoint. The status board gets its standup entry. +- [ ] **Step 7 — run everything.** `just wovm-test`, `just residency`, + `just db-actor`, `just linkcheck`. All green. +- [ ] **Step 8 — commit.** Prefix `feat(db2-delta)`. + +--- + +## What this plan deliberately does not do + +- **No chain cap and no read-triggered fold.** Settled in the spec: compaction + is the bound, and a cap is a number nobody would tune. +- **No measurement.** Whether this mode beats the kernel's own paging is + databasev2 2's task 7, and it decides whether any of this is worth keeping. + This plan must not be read as answering it. +- **No change to `resident: all`.** If a diff touches that path, it is wrong. + +## The risk to watch while executing + +The fold is written once and called from three places. The tests are arranged so +each caller is proven separately — Task 2 for reads, Task 5 for replay and +compaction, Task 6's oracle for all of them against ordinary storage. If a task +finds itself wanting a second fold "just for this caller", that is the design +failing and it should stop and say so rather than write it.