docs(db2-keys): spec — delta records for keys-resident updates
- updates append a DELTA (class, id, field, value, back-pointer), not a full row. The workload decides it: a product catalogue changes one narrow field of a wide row on every order, so a full-row append would rewrite every field to move one integer on a shop's hottest path - the back-pointer keeps the id map at one slot per row, which is the mode's whole premise; a map growing per update would defeat it - ONE fold function, three callers (read, replay, compaction). Three implementations of one rule is how they drift, and a fold that differs between reading and replaying is a database that changes its mind at boot. Named as the design's principal risk - indexed columns MAY change: price is exactly what a catalogue indexes, so forbidding it would be a restriction users meet immediately - no chain cap, deliberately. Compaction already rewrites live rows, so every checkpoint resets every chain, and deltas grow the log which pulls the next checkpoint forward — the workload that lengthens chains triggers the fold that shortens them - the risk that accepts: one hot SKU under an otherwise quiet write rate. Task 7's benchmark must include it - supersedes the 2026-08-26 spec's one-line full-row Update sketch, marked in place rather than left as a second design in the tree Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit c9a88b05c4c62a5df13253686c990aaccc3f9cbb)
This commit is contained in:
parent
0b0d54121f
commit
04017aea26
2 changed files with 179 additions and 2 deletions
|
|
@ -124,8 +124,15 @@ indexes are very much resident.
|
|||
each field. Both functions already exist in `wal.c` and are already exercised
|
||||
by replay; the change is that they are called on demand rather than only at
|
||||
boot.
|
||||
- **Update** — append a new record, repoint the offset. The superseded record
|
||||
becomes garbage, reclaimed by the checkpoint.
|
||||
- **Update** — ~~append a new record, repoint the offset~~. **SUPERSEDED
|
||||
2026-08-30** by
|
||||
[`2026-08-30-keys-resident-delta-updates-design.md`](2026-08-30-keys-resident-delta-updates-design.md).
|
||||
This line was written before any of the mode was built and describes a
|
||||
full-row append, which the motivating workload rejects: a product catalogue
|
||||
changes one narrow field of a wide row on every order, so a full-row append
|
||||
rewrites every field to move one integer. Updates append a **delta** carrying
|
||||
a back-pointer, folded on read, with compaction doing the fold that keeps
|
||||
chains short. The superseded-record-becomes-garbage half is still true.
|
||||
- **Delete** — append a tombstone, drop the id from the map and every index.
|
||||
- **Scan** — a sequential walk of the log, which is the case log-structured
|
||||
storage is best at. Cost changes from memory-speed to sequential-disk; the
|
||||
|
|
|
|||
|
|
@ -0,0 +1,170 @@
|
|||
# `resident: keys` updates — delta records folded on read
|
||||
|
||||
Design settled 2026-08-30. Closes the last gap in
|
||||
[databasev2 2](../../stories/databasev2/02-table-storage-modes.md), which is
|
||||
the only reason the loader still refuses `resident: keys` outright.
|
||||
|
||||
**Supersedes** the one-line sketch in
|
||||
[`2026-08-26-table-residency-design.md`](2026-08-26-table-residency-design.md)
|
||||
— "Update: append a new record, repoint the offset". That was written before
|
||||
any of the mode was built, and it describes the full-row shape this design
|
||||
rejects.
|
||||
|
||||
## The workload that decides the shape
|
||||
|
||||
The docs used to motivate this mode with a 120 GB audit table. That was a bad
|
||||
example, because an audit log is append-only and therefore argues for nothing:
|
||||
if rows are never updated, the missing update path costs nothing and the mode
|
||||
could simply have been declared append-only.
|
||||
|
||||
The real case is a **product catalogue**, and it is the one in
|
||||
`docs/examples/residency`. Its shape decides everything below:
|
||||
|
||||
- The id is stable. A SKU does not change.
|
||||
- Other fields change constantly. Every order moves `stock`.
|
||||
- Reads are hot. Browsing a catalogue is the dominant operation.
|
||||
- Rows are wide. Name, price, description, and whatever else a shop carries.
|
||||
- The table is the one that outgrows RAM first, which is why it wants this mode.
|
||||
|
||||
So an update touches **one narrow field of a wide row, frequently, on the
|
||||
hottest write path a shop has**, while reads must stay fast. Appending the whole
|
||||
row per sale would rewrite every field to move one integer. That is the argument
|
||||
for a delta.
|
||||
|
||||
## The design
|
||||
|
||||
### The record
|
||||
|
||||
A fourth WAL record kind beside `WO_WAL_INSERT` (1), `WO_WAL_REMOVE` (2) and
|
||||
`WO_WAL_UPDATE` (3). It carries the class id, the row id, the field index, the
|
||||
encoded new value, and a **back-pointer**: the log offset of the record this one
|
||||
supersedes.
|
||||
|
||||
The back-pointer is what keeps the id map at exactly one slot per row. That is
|
||||
not an optimisation — a map that grew per update would defeat the mode's entire
|
||||
premise, which is that the map is the only resident part.
|
||||
|
||||
### The fold, and why it lives in one function
|
||||
|
||||
Reading a row means walking the chain **backward** from the newest record,
|
||||
taking first-seen-wins per field, until a full-row record terminates it, then
|
||||
materialising that base and overlaying the collected fields.
|
||||
|
||||
Three callers need exactly this: reads (`wo_row_borrow`), replay, and
|
||||
compaction. They must agree perfectly — a fold that differs between reading and
|
||||
replaying is a database that changes its mind at boot. **This is the design's
|
||||
principal risk**, and the mitigation is that the fold is written once and all
|
||||
three call it. No caller may implement its own walk.
|
||||
|
||||
### Update
|
||||
|
||||
Borrow the row, which folds it. That read is needed regardless: index
|
||||
maintenance requires the old value, and this design allows indexed columns to
|
||||
change (see below). Then append the delta, commit, and re-point the map at the
|
||||
new record.
|
||||
|
||||
The two existing entry points — the VM-value path and the engine-value slot
|
||||
path — converge on one internal append. They currently both refuse keys tables;
|
||||
that refusal is replaced, not duplicated.
|
||||
|
||||
### Indexed columns may change
|
||||
|
||||
Allowed deliberately. On a product table `price` is exactly the kind of column
|
||||
you would index, so forbidding it would be a restriction users meet immediately
|
||||
rather than a theoretical one.
|
||||
|
||||
The cost is that index maintenance runs on every delta: the old value comes from
|
||||
the row just folded, the new one from the delta itself. `@unique` probing
|
||||
already borrows each bucket candidate, and those borrows now fold — so a unique
|
||||
check against a heavily-updated table walks chains. Disclosed, not hidden.
|
||||
|
||||
### Crash safety
|
||||
|
||||
Commit precedes re-point, and that ordering is the whole argument:
|
||||
|
||||
- Crash **after** the barrier, before the re-point: the delta is durable, the
|
||||
in-memory map is stale, and replay rebuilds the map from the log — which sees
|
||||
the delta and lands on the same state. Correct.
|
||||
- Crash **before** the barrier: the delta never reached the log and the row is
|
||||
unchanged. Correct.
|
||||
|
||||
Under group commit an update stages like an insert, and the re-point defers to
|
||||
the post-barrier flush — a pending-repoint list mirroring the pending-drop list
|
||||
that already exists in `wo_wal` (`pend`/`pend_len`/`pend_cap`).
|
||||
|
||||
### Compaction is the bound
|
||||
|
||||
Compaction already walks live ids and rewrites each row. It now **folds** each
|
||||
row and writes a single full-row record, so every checkpoint resets every chain
|
||||
to length zero.
|
||||
|
||||
This is what makes read cost bounded without a cap: a read is
|
||||
`1 + deltas-since-the-last-compaction` preads, and deltas grow the log, which
|
||||
pulls the next compaction forward on the existing `WO_CHECKPOINT_BYTES` and
|
||||
`WO_CHECKPOINT_RATIO` policy. The workload that lengthens chains is the same one
|
||||
that triggers the fold that shortens them.
|
||||
|
||||
**Settled deliberately: no hard chain cap.** The self-limiting property above is
|
||||
real, and a cap is a number nobody would tune. The risk this accepts is stated
|
||||
under Risks.
|
||||
|
||||
### Replay
|
||||
|
||||
A delta at replay re-points the row's offset, exactly as a dropped payload
|
||||
does today. Because indexed columns may change, replay must also fold the delta
|
||||
into the indexes it built from the base record — so boot work grows with chain
|
||||
length. Chains are short by the compaction argument, and this is the same fold
|
||||
function reads use.
|
||||
|
||||
## What this does NOT change
|
||||
|
||||
- The id map's shape — one offset per row, unchanged.
|
||||
- The insert path, the read path for unmodified rows, deletes, scans, or
|
||||
checkpoint survival. All of those shipped and are tested.
|
||||
- `resident: all`. Nothing here touches the fully-resident path.
|
||||
|
||||
## Risks
|
||||
|
||||
- **The three folds drifting apart.** Mitigated by one function, and the
|
||||
acceptance criteria below test the same row through all three paths rather
|
||||
than testing the fold once.
|
||||
- **One hot row under an otherwise quiet write rate.** This is the shape that
|
||||
breaks the compaction argument: a single SKU selling far faster than the rest
|
||||
lengthens its chain without growing the log enough to trigger a checkpoint.
|
||||
Deliberately not solved with a cap. **Task 7's benchmark must include it**,
|
||||
because it is the case where this design is worst and it is a realistic
|
||||
inventory shape, not a contrived one.
|
||||
- **Unique probing on a hot table.** Each bucket candidate now folds. Bounded by
|
||||
the same checkpoint argument, but it multiplies: candidates times chain length.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **Given** a keys-resident row, **when** one field is updated, **then** a read
|
||||
returns the new value for that field and the original values for every other.
|
||||
- **Given** a row updated N times, **when** it is read, **then** the result
|
||||
equals the same row's value in a fully-resident table subjected to the same
|
||||
updates. The resident table is the oracle.
|
||||
- **Given** a row with a delta chain, **when** the process restarts, **then**
|
||||
replay produces the same row the pre-restart read produced.
|
||||
- **Given** a row with a delta chain, **when** compaction runs, **then** the row
|
||||
reads identically afterwards and its chain is length zero.
|
||||
- **Given** a delta that changes an indexed column, **when** the row is queried
|
||||
through that index, **then** it is found by the new value and not by the old —
|
||||
before and after a restart, and before and after a compaction.
|
||||
- **Given** a crash between the commit barrier and the re-point, **when** the
|
||||
process restarts, **then** the update is present.
|
||||
- **Given** an update to a `resident: all` table, **when** it runs, **then**
|
||||
nothing about its behaviour or its log records changes.
|
||||
- **Given** `resident: keys` on a table, **when** the program loads, **then** it
|
||||
is **accepted** — the loader refusal is lifted by this work, and that lift is
|
||||
the deliverable.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- A hard delta-chain cap, and any read-triggered fold. Settled above.
|
||||
- Changing the compaction policy knobs. They already exist and this design
|
||||
leans on them unchanged.
|
||||
- `resident: keys` performance measurement. That is
|
||||
[databasev2 2](../../stories/databasev2/02-table-storage-modes.md)'s task 7,
|
||||
and it decides whether this mode is worth having at all — a question this
|
||||
design does not answer and must not be read as answering.
|
||||
Loading…
Reference in a new issue