shoney/writeonce
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(db2-delta): implementation plan for keys-resident delta updates

Browse source 
- 6 tasks: the record kind, the fold, updates + indexes, the request
  path and group commit, replay + compaction, then lifting the loader
  refusal and proving it end to end
- the fold is written ONCE and called from three places; the tests are
  arranged to prove each caller separately, and the plan says that
  wanting a second fold "just for this caller" means the design failed
- Task 2 includes a same-field ordering test specifically, because a
  fold walking the chain backwards the wrong way returns plausible data
  and is otherwise invisible
- Task 6 step 1 audits the db.c request arms BEFORE lifting anything —
  they were never audited for keys-residency the way the inline path
  was, and the last audit of that kind found delete corrupting memory
- self-review found the spec's crash criterion had no task: added a step
  that commits a delta, skips the re-point, and replays, which is the
  state a crash between barrier and flush leaves behind
- every symbol the plan names verified to exist in database/src
- code-free per house convention; the writing-plans skill wants code
  blocks and the project rule overrides it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit abb8fc9454bdca9d8d703c8cfeb224f89a93d14d)
This commit is contained in:
shoney.arickathil 2026-08-30 06:34:21 +02:00
parent 04017aea26
commit 3f88b07abb
1 changed files with 303 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

303
docs/superpowers/plans/2026-08-30-keys-resident-delta-updates.md Normal file
View file

@ -0,0 +1,303 @@
# `resident: keys` delta updates — implementation plan
> **For agentic workers:** REQUIRED SUB-SKILL: use
> `superpowers:subagent-driven-development` or `superpowers:executing-plans`,
> task by task. Steps are checkboxes.
>
> **House convention: this plan carries concept, reason and actions in words —
> no implementation or test code blocks.** The engineer writes the code; the
> plan says what must be true and why, and names every file and symbol
> involved. Exact values that must match are quoted inline.
**Goal:** let a row on a `resident: keys` table be updated, so the loader's
refusal of that annotation can be lifted.
**Architecture:** an update appends a **delta** record — class, id, field index,
new value, and a back-pointer to the record it supersedes — instead of a full
row. Reading walks the chain backward, first-seen-wins per field, until a
full-row record terminates it. Compaction folds every chain flat, so read cost
is bounded by updates since the last checkpoint rather than by a row's lifetime.
**Tech stack:** C11, the embedded database under `database/src/` (`wal.c`,
`table.c`, `db.c`), the runtime loader (`runtime/src/loader.c`), the C test
suite (`runtime/test/test_wal.c`), and the `.wo` example under
`docs/examples/residency` gated by `scripts/residency-accept.sh`.
**Spec:**
[`2026-08-30-keys-resident-delta-updates-design.md`](../specs/2026-08-30-keys-resident-delta-updates-design.md)
## Global constraints
- **One fold, three callers.** Reads, replay and compaction must fold
identically. Write it once; no caller may implement its own walk. The spec
names this the design's principal risk: a fold that differs between reading
and replaying is a database that changes its mind at boot.
- **Commit before re-point, always.** A crash after the barrier leaves a
durable delta and a stale map, which replay reconciles. A crash before leaves
the row untouched. Reversing the order breaks both.
- The id map keeps **exactly one slot per row**. A map that grew per update
would defeat the mode's premise, which is that the map is the only resident
part.
- Indexed columns **may** change in a delta. Index maintenance runs per delta.
- Never delete-then-insert as an update. Assigning to a row field writes
through and maintains indexes.
- `resident: all` behaviour and its log records must not change at all.
- Every task ends green on `just wovm-test`; the final tasks also on
`just residency` and `just linkcheck`.
- Commits: bullets only, no prose paragraphs, at most 25 lines, title prefixed
`feat(db2-delta): ` or `fix(db2-delta): `, ending with exactly
`Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>`.
Stage explicit paths; never `git add -A`. Do not push, branch or merge.
## File structure
| File | Responsibility |
| --- | --- |
| `database/src/wal.h` | the record-kind enum gains a fourth value; the pending-repoint list declaration |
| `database/src/wal.c` | encode/append a delta, **the one fold function**, replay's delta arm, compaction's fold-and-flatten |
| `database/src/table.c` | the update entry points stop refusing and call the append; index maintenance across a delta |
| `database/src/db.c` | the two request-path update arms stage a delta and defer the re-point |
| `runtime/src/loader.c` | the refusal is lifted — the deliverable |
| `runtime/test/test_wal.c` | the C-level proofs, including the resident-table oracle |
| `docs/examples/residency/` + `scripts/residency-accept.sh` | the end-to-end proof that only becomes possible once the refusal lifts |
The fold lives in `wal.c` because that is where records are read; `table.c`
calls it through the existing `wo_row_borrow`.
---
### Task 1 — the record kind and its encoding
**Files:** modify `database/src/wal.h` (the enum at line 46,
`WO_WAL_INSERT = 1, WO_WAL_REMOVE = 2, WO_WAL_UPDATE = 3`) and
`database/src/wal.c`.
**Produces:** `WO_WAL_DELTA = 4`, and an internal append that stages one delta
record. A delta's payload is: kind byte, class id, row id, field index, the
back-pointer offset, then the field's value encoded by the existing `enc_val`.
- [ ] **Step 1 — extend the enum.** Add the fourth value. Do not renumber the
existing three: they are on disk in every log that exists.
- [ ] **Step 2 — write the failing test.** In `runtime/test/test_wal.c`, stage a
delta, commit, then read the raw record back with `scan_record` and assert
every field round-trips: kind, class, id, field index, back-pointer, and
the value. It must fail before the encoder exists.
- [ ] **Step 3 — run it and watch it fail.** `make -C runtime build/test_wal &&
./runtime/build/test_wal`. Expect a compile failure naming the missing
symbol — that is a legitimate first failure for C.
- [ ] **Step 4 — implement the append.** Follow `wo_wal_append_insert`'s shape:
build a `wbuf`, `wput_*` the header fields, `enc_val` the one value, and
`stage()` it. Take the back-pointer as a parameter rather than reading it
from the map — the caller knows it and this keeps the encoder ignorant of
table state.
- [ ] **Step 5 — run the test.** Expect pass.
- [ ] **Step 6 — commit.** Prefix `feat(db2-delta)`.
---
### Task 2 — the fold, and reads through it
**Files:** modify `database/src/wal.c` (near `wo_wal_read_row_at`, line ~600)
and `database/src/table.c` (`wo_row_borrow`).
**Consumes:** `WO_WAL_DELTA` and the record layout from Task 1.
**Produces:** the single fold function every later task calls. Given a starting
offset it walks back through delta records, remembering the first value seen for
each field index, until it meets an INSERT or UPDATE record; it materialises
that base row and overlays the remembered fields. Name it once and do not
duplicate it.
- [ ] **Step 1 — write the failing test.** Insert a keys-resident row, drop its
payload, append two deltas to two *different* fields, re-point after each,
then borrow the row and assert both changed fields carry their new values
and the untouched fields carry their originals.
- [ ] **Step 2 — write a second failing test that pins the ordering rule.** Two
deltas to the SAME field. The newer must win. This is the test that
catches a fold walking the chain in the wrong direction, which is
otherwise invisible — a wrong-direction fold returns plausible data.
- [ ] **Step 3 — run both, watch them fail.**
- [ ] **Step 4 — implement the fold.** Walk backward from the given offset.
Track which field indexes have been resolved; skip a delta for a field
already resolved, because the newer one was seen first. Stop at the first
full-row record and decode it, then overlay. Guard against a cycle in the
back-pointers by refusing to walk more records than the log can hold —
corrupt back-pointers must fail loudly, not spin.
- [ ] **Step 5 — route `wo_row_borrow` through it.** Its keys arm currently
calls `wo_wal_read_row_at` directly; it now folds instead. Everything the
borrow already guarantees stays: the per-table scratch, the `scratch_busy`
nested-borrow refusal, and the cid/id identity check.
- [ ] **Step 6 — run the whole WAL suite.** Every existing keys test must still
pass — a row with no deltas folds to itself, which is the case they cover.
- [ ] **Step 7 — commit.** Prefix `feat(db2-delta)`.
---
### Task 3 — updates append, indexes follow
**Files:** modify `database/src/table.c` (`wo_row_update_field` and
`wo_row_update_field_slot` — both currently refuse keys tables with
"update on a `resident: keys` table is not implemented") and
`database/src/wal.c`.
**Consumes:** the append from Task 1, the fold from Task 2.
**Produces:** a working update on a keys-resident table, and the pending-repoint
list the request path needs in Task 4.
- [ ] **Step 1 — write the failing test.** Update one field of a keys-resident
row through `wo_row_update_field`, then read it back and assert the new
value. It fails today with the refusal message.
- [ ] **Step 2 — write the index test, which is the one that matters.** Give the
table a secondary index, update an **indexed** column, then query through
that index and assert the row is found by its NEW value and NOT by its
old. The spec allows indexed columns to change specifically because a
catalogue indexes `price`, so this is the realistic case, not an edge one.
- [ ] **Step 3 — run both, watch them fail.**
- [ ] **Step 4 — replace the refusals.** Both entry points converge on one
internal path: borrow the row (which folds), read its current offset with
`wo_row_offset1`, append the delta with that offset as the back-pointer,
commit, then `wo_row_set_offset` to the new record.
- [ ] **Step 5 — maintain the indexes.** The borrowed row is the OLD row; call
`idx_remove_row` with it, apply the field change to the materialised copy,
then `idx_add_row`. Both functions already exist in `table.c` (lines 452
and 487) and both take a `db_row *`.
- [ ] **Step 6 — release the borrow on every exit**, including the failure
arms. A keys borrow holds the per-table scratch and blocks the next
borrow until released; an early return that skips it deadlocks the table.
- [ ] **Step 7 — run the suite.** Expect pass.
- [ ] **Step 8 — commit.** Prefix `feat(db2-delta)`.
---
### Task 4 — the request path and group commit
**Files:** modify `database/src/db.c` (the two `WO_B_DB_UPDATE_FIELD` arms at
lines 88 and 289 — one inline, one on the request path) and
`database/src/wal.c` (the pending list beside `pend`/`pend_len`/`pend_cap` in
`wal.h`, and `wo_db_flush_drops`).
**Consumes:** the update path from Task 3.
- [ ] **Step 1 — read how inserts already do this before writing anything.**
The insert path stages, lets the drain commit once at a barrier, and only
then applies its map change through `wo_db_flush_drops`. An update must
follow the same shape: the re-point is a map change that must not happen
before the barrier.
- [ ] **Step 2 — write the failing test.** Two updates to the same row inside
one drain, then a read. Both must be visible, and the row's chain must
have both deltas. A re-point applied too early shows up as the second
delta's back-pointer skipping the first.
- [ ] **Step 3 — run it, watch it fail.**
- [ ] **Step 4 — add the pending-repoint list.** Mirror the pending-drop list's
declaration and lifetime exactly. Do not reuse the drop list: a drop and a
re-point carry the same three fields but mean different things, and
merging them would make a future reader guess.
- [ ] **Step 5 — wire both arms.** The inline arm commits and flushes
immediately, as it does for inserts; the request arm records the pending
re-point and lets the drain's barrier flush it.
- [ ] **Step 6 — run the suite, and `just db-actor`** — that gate drives actors
on worker shards through the request path, which is the arm most easily
got wrong.
- [ ] **Step 7 — commit.** Prefix `feat(db2-delta)`.
---
### Task 5 — replay and compaction fold the same way
**Files:** modify `database/src/wal.c` (`apply_record` at line 738,
`wo_wal_replay_ex` at line ~861, and `wo_wal_compact`).
**Consumes:** the fold from Task 2.
- [ ] **Step 1 — write the failing replay test.** Build a row with a chain of
three deltas, close the log, replay it into a fresh database, and assert
the row reads exactly as it did before the restart. Assert the indexes too:
query through a secondary index whose column one of the deltas changed.
- [ ] **Step 2 — write the failing compaction test.** Same chain, then compact,
then assert the row still reads identically **and its chain is now length
zero** — the record its offset points at must be a full row, not a delta.
That second assertion is the one proving compaction is the bound the spec
relies on; without it the test passes even if compaction copied the chain.
- [ ] **Step 3 — run both, watch them fail.**
- [ ] **Step 4 — replay's delta arm.** A delta re-points the row's offset, the
same way a dropped payload does. Because indexed columns may change,
replay must also fold the delta into the indexes built from the base
record. Note `wo_wal_replay_ex` already lends the runtime a read-only WAL
view for the replay's duration — the fold needs it and it is already there.
- [ ] **Step 5 — compaction folds flat.** It already walks live ids and copies
each row's current record. It must now fold each row and write ONE
full-row record, so every checkpoint resets every chain. Keep the existing
byte-for-byte copy for rows whose chain is already empty: it is faster and
it is what the current code does.
- [ ] **Step 6 — the crash window, which is the criterion nothing else covers.**
Append a delta, commit it, and then do NOT re-point the map — that is
exactly the state a crash between the barrier and the flush leaves behind.
Replay the log into a fresh database and assert the update IS present.
This is the test proving the commit-before-re-point ordering earns its
place; without it the ordering is an untested assertion in a comment.
`test_compact_crash_battery` in the same file shows the established shape
for simulating a crash point without actually killing a process.
- [ ] **Step 7 — run the suite plus `just residency`.**
- [ ] **Step 8 — commit.** Prefix `feat(db2-delta)`.
---
### Task 6 — lift the refusal, and prove it end to end
**Files:** modify `runtime/src/loader.c` (the `WO_CLASSF_RESIDENT_KEYS` BAIL),
`docs/examples/residency/main.wo`, `docs/examples/residency/README.md`,
`scripts/residency-accept.sh`, `docs/stories/databasev2/02-table-storage-modes.md`,
`docs/stories/00-status.md`.
**Consumes:** everything above.
- [ ] **Step 1 — audit the request path before lifting anything.** The `db.c`
request arms were never audited for keys-residency the way the inline
path was, and the last audit of that kind found `delete` corrupting
memory. Read every arm that touches a row on a keys table and confirm it
goes through borrow/release or `wo_row_next_id`, never `wo_row_ptr`
directly. Report what you checked, not just the conclusion.
- [ ] **Step 2 — lift the loader refusal.** Delete the BAIL. Keep the second
one: `durable: false` with `resident: keys` stays refused, because a row
that is never logged has nowhere to be read from.
- [ ] **Step 3 — make the example real.** In `docs/examples/residency/main.wo`,
uncomment the keys-resident `Product` declaration and make it the live
one. `place_order` already updates `stock`, which is now the feature under
test rather than the thing that cannot work.
- [ ] **Step 4 — gate it.** `scripts/residency-accept.sh` currently asserts the
annotation is REFUSED at load. That leg must invert: it now asserts the
program runs and the order decrements stock across a restart. Delete the
refusal assertion rather than leaving it inverted-but-present.
- [ ] **Step 5 — the oracle test.** In `test_wal.c`, run the same sequence of
updates against a `resident: all` table and a `resident: keys` table and
assert the rows are identical at every step. The resident table is the
oracle; this is the strongest available check that the fold agrees with
ordinary storage.
- [ ] **Step 6 — update the docs to say what is now true.** The story's
Outstanding criterion for updates moves to Met, recording HOW it was
verified. The example's README loses its "why it is refused" section and
gains what the mode costs: a read is one pread plus deltas since the last
checkpoint. The status board gets its standup entry.
- [ ] **Step 7 — run everything.** `just wovm-test`, `just residency`,
`just db-actor`, `just linkcheck`. All green.
- [ ] **Step 8 — commit.** Prefix `feat(db2-delta)`.
---
## What this plan deliberately does not do
- **No chain cap and no read-triggered fold.** Settled in the spec: compaction
is the bound, and a cap is a number nobody would tune.
- **No measurement.** Whether this mode beats the kernel's own paging is
databasev2 2's task 7, and it decides whether any of this is worth keeping.
This plan must not be read as answering it.
- **No change to `resident: all`.** If a diff touches that path, it is wrong.
## The risk to watch while executing
The fold is written once and called from three places. The tests are arranged so
each caller is proven separately — Task 2 for reads, Task 5 for replay and
compaction, Task 6's oracle for all of them against ordinary storage. If a task
finds itself wanting a second fold "just for this caller", that is the design
failing and it should stop and say so rather than write it.