docs(db2-migrate): close out iteration 12

- crash-before-rename test: a COMPLETE valid migrated temp beside the
  untouched original is discarded and the boot re-migrates — the
  sharpest point on the crash timeline, deterministic, no fault
  injection needed
- story: all six tasks done with commit hashes, all eight criteria met
  with the test that proves each, plus the three deviations from the
  plan and why (transcode over replay, lazy head, poison forces
  transcode)
- CODE-LOGIC: migration section; also corrected limitation 3, which
  still claimed unbounded hot-row chains — iteration 11 closed that
- status board row 12; deploy guide's rollback section gets its real
  answer (rolling back across a migration is a migration backwards:
  expect the refusal, restore the .bak)
- test_wal 5966 pass, 0 fail

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 4bb6ece2531e2123eb958c91d9bef4a6528eab3b)
This commit is contained in:
shoney.arickathil 2026-08-31 21:47:08 +02:00
parent 27aecd3dc1
commit 4ad24d6381
5 changed files with 157 additions and 36 deletions

View file

@ -319,8 +319,44 @@ the `db_text*` layout) before the fix, pinned by
2. *Replay is O(N²) in a row's delta-chain length* — `apply_delta` folds the
pre-delta row, and `wo_row_remove` (called internally) folds the SAME
offset again, so each replayed delta re-walks its whole chain.
3. *Compaction triggers on byte ratio only* — `wo_wal_should_compact` has no
per-row delta-count signal, so one hot row (a single popular SKU) can grow
a long personal chain without moving the aggregate ratio enough to fire a
checkpoint. The no-chain-cap design decision rests on compaction bounding
length; for this shape it does not.
3. *Compaction triggers on byte ratio only* — **closed by databasev2 11**:
the fold reports hop count and `row_apply_field_keys` writes a full-row
image (`WO_WAL_UPDATE`) past `WO_DELTA_MAX_HOPS` (16), so a hot row's
chain is bounded in the update path itself; the checkpoint no longer
carries that burden. `wo_wal_should_compact` also gained an absolute
garbage term (`WO_CKPT_ABS_BYTES`).
## Schema migrations (databasev2 12)
A `@table` class is the schema; the log is the database; boot compares them.
- **The log describes itself.** `WO_WAL_SCHEMA` (kind 5) is the head record
of every fresh and every compacted log: per class its NAME, storage flags,
and per field name + kind + the two encoding-relevant metadata words.
Written lazily by `stage()` ahead of the FIRST real record — never for a
log that stays empty, because `durable: false` programs have a documented
zero-bytes contract. `apply_record` skips it before reading cid/id (its
class count would be misread as a cid); replay does not count it.
- **The diff is name-keyed** (`wo_schema_diff`). Classes match by name,
fields by name + kind, owned references (`fclass`) by the NAME the number
resolves to — so pure declaration reordering costs only a cid remap, which
closes the old silent hole where reordering decoded rows into the wrong
class. Verdicts are per-class POISONS carried in the plan: retype,
same-shape delete+add (a disguised rename), vanished class, storage-flag
change, and the embed closure (any class whose stored values carry a
CHANGED class's old sub-shape, to a fixpoint). A poison forces the
transcode and bites only when a record of the class is actually met — no
rows, no verdict.
- **The migration is a record-level transcode** (`wo_wal_migrate`), not a
replay: no id maps, no indexes, no keys-resident logic. Old shapes decode
through a classdesc shim built from the stored schema; embedded cids are
renumbered by `mig_fixup_cids` (owned values carry a cid on the wire);
surviving fields move slots, deleted values are freed, added fields take
`enc_val(0)` — the kind's zero. Delta back-pointers rewrite through an
offset map, and a delta on a deleted field is SPLICED: it maps to its own
target, so later deltas step over it. Temp + fsync + rename, compaction's
own crash discipline — a kill anywhere leaves the old log authoritative,
including a kill after the temp is complete (`test_migrate_crash_before_rename`).
- **Legacy logs** (no head record) replay exactly as before and adopt the
head at their next compaction. v1 verbs are add and delete only; rename
wants `@renamed_from` (v2), data/seed migrations are v2.

View file

@ -205,9 +205,14 @@ rm -rf /srv/writeonce-site/data && mv /srv/writeonce-site/data.bak-<date> /srv/w
systemctl start writeonce-site
```
An older binary against a newer WAL is fine here only because the `Chapter`
schema has not changed. Once it does, that assumption dies and this section
needs a real answer.
Schema changes are handled since databasev2 12: a binary whose `@table`
classes gained or lost fields migrates the WAL at startup (the log's head
record carries the shape that wrote it), and an incompatible change — a
retyped field, a vanished class — refuses to start by name instead of
reporting corruption. Rolling BACK across a migration is itself a schema
change in the other direction: the old binary predates the head record's
shape, so expect the same refusal — restore the `.bak` data directory
alongside the old binary rather than pointing it at migrated data.
## Known gaps

View file

@ -996,7 +996,7 @@ the language arc as v1 history.
| 9 | [Cross-program tables](databasev2/09-cross-program-tables.md) *(was 20)* | ⏸ hold — attach to a running program's database over local IPC |
| 10 | [Keypair attach auth](databasev2/10-keypair-attach-auth.md) *(was 21)* | ⏸ hold — program identity as a keypair; needs 9 |
| 11 | [Bounded delta chains](databasev2/11-bounded-delta-chains.md) | ✅ **LANDED 2026-08-30.** A `resident: keys` row's delta chain is bounded in the UPDATE path, because the checkpoint is blind to per-row chain length — it thresholds on whole-log bytes, so one hot row can grow an unbounded chain inside a log that never trips compaction. The fold now reports hop count (free — the walk already visited every hop), and past `WO_DELTA_MAX_HOPS` (16) the update writes a full row image instead of a delta, resetting depth to 0. **Two things the tests corrected.** The flattened image is a `WO_WAL_UPDATE`, not an `INSERT`: the row's original INSERT is already in a live log, so a second one for the same id is a duplicate that replay correctly refuses as corruption — INSERT is right only for compaction, which builds a *fresh* log. And the **proportional ceiling was removed as dead code**: with the absolute term at 64 MiB, garbage large enough to reach a 256 MiB ceiling has already tripped it, so the branch was unreachable. Borrowing both constants from postgres was the wrong inference — PG needs two because it thresholds on *tuples* with its pair at opposite ends (base 50, max 1e8); this thresholds on *bytes*, where one constant does both jobs. Found by trying to write a test for the ceiling and finding no input could reach it. Four tests: depth stays bounded across 2K+2 updates, a flattened chain replays, a delta on an **indexed** column composes with flattening (checked at every step across the bound and after restart — found no product defect), and the policy's absolute term with its boundary. `test_wal` **5700 pass / 0 fail**; `wovm-test` and `woc-test` green. **One criterion is weaker than written:** the replay check asserts an expected value, not a `resident: all` oracle table. [spec](../superpowers/specs/2026-08-30-bounded-delta-chains-design.md) |
| 12 | [Schema migrations](databasev2/12-schema-migrations.md) | 🔄 **in progress (brainstormed + spec'd 2026-08-31).** A `@table` class is the schema, the log is the database, and today nothing compares them: an added or deleted field turns a healthy `WO_DATA` into "corruption" at boot, and reordering declarations silently decodes rows into the wrong class (records name classes by declaration index). v1, locked: **declarative and automatic at boot** — a `WO_WAL_SCHEMA` record at the log head states the shape; boot diffs it against the compiled classes by NAME; add and delete migrate through a compaction-style rewrite (added fields zero-filled — the grammar has no default syntax; deleted values freed at the rename swap); retype, same-kind delete+add (a disguised rename), and vanished classes REFUSE by name. Data/seed migrations deferred to v2. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) |
| 12 | [Schema migrations](databasev2/12-schema-migrations.md) | ✅ **LANDED 2026-08-31.** A `@table` class is the schema, the log is the database, and boot now compares them — before this, an added or deleted field turned a healthy `WO_DATA` into "corruption" and reordering declarations silently decoded rows into the wrong class. Landed: `WO_WAL_SCHEMA` head record (written LAZILY ahead of the first real record — an eager head broke `durable: false`'s documented zero-bytes contract by 75 bytes and the gate caught it), a name-keyed diff whose refusals are per-class POISONS that bite only when a record of the class is met, and a record-level TRANSCODE: cids remap by name including inside stored owned values, deleted values freed, added fields zero-filled, delta back-pointers rewritten through an offset map with deltas on deleted fields SPLICED out; temp+fsync+rename, compaction's crash discipline. **Two bugs the tests forced out:** a poisoned class skipped plan identity so the retype refusal fell through to generic "corruption" (the message this iteration exists to replace), and early `goto corrupt` freed uninitialized memory. End-to-end: `migrating \`Note\`: +flag` then `flag=0`; retype refuses naming `val`, exit 2, old binary still boots the refused log. 21 new tests, `test_wal` **5966/0**; wovm/woc/site/residency gates green. v2 holds rename (`@renamed_from`), retypes, and data/seed migrations. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) |
---

View file

@ -1,7 +1,7 @@
---
track: databasev2
iteration: "12"
status: in-progress
status: done
readiness: ready
---
@ -36,35 +36,55 @@ readiness: ready
| # | Task | State |
| --- | --- | --- |
| 1 | `WO_WAL_SCHEMA` record: encode/decode, emitted by fresh-log open and by compaction | ⬜ |
| 2 | boot diff: peek head, match by class/field NAME, classify migrate / refuse / legacy | ⬜ |
| 3 | migration pass: old-shape replay through a field map, compaction-style rewrite, rename swap | ⬜ |
| 4 | refusals: retype, same-kind delete+add, vanished class — house-style messages | ⬜ |
| 5 | tests: the acceptance list, incl. keys-resident, legacy log, crash injection | ⬜ |
| 6 | close out: story, status board, deploy-guide note | ⬜ |
| 1 | `WO_WAL_SCHEMA` record: encode/decode, written LAZILY ahead of the first record and by compaction | ✅ `ba8519f` |
| 2 | boot diff: peek head, match by class/field NAME, classify migrate / refuse / legacy | ✅ `63a063b`, wired `b21943a` |
| 3 | migration pass: a record-level TRANSCODE (no replay, no db state), rename swap | ✅ `b69092a` |
| 4 | refusals: per-class poisons that bite only when a record of the class is met | ✅ `63a063b` + `b21943a` |
| 5 | tests: 21 new across three tiers, incl. delta splice, owned cid fixup, valid-temp crash | ✅ `test_wal` 5966/0 |
| 6 | close out: story, status board, deploy-guide note, CODE-LOGIC | ✅ this commit |
## What changed against the plan, and why
- **The migration pass became a record-level transcode**, not an old-shape
replay: the log is rewritten record by record (cids remapped by name,
fields moved/dropped/zero-filled, delta back-pointers rewritten through an
offset map) and the result replays through machinery that already exists
and is already tested. No id maps, no indexes, no keys-resident logic in
the migration itself.
- **The head record is written lazily**, ahead of the first real record. The
eager version broke `durable: false`'s documented zero-bytes contract by
75 bytes; `residency-accept` caught it.
- **A poison forces the transcode instead of being skipped.** The first
end-to-end retype fell through to replay's generic "corruption" because a
poisoned class did not break plan identity — the exact message this
iteration exists to replace. Caught by the language-level smoke test.
## Acceptance criteria
- **Given** a log written with shape A and a binary with one added field,
**when** it boots, **then** it runs, the field reads zero-valued, and the
log head carries the new schema.
- **Given** one deleted field, **when** it boots, **then** dropped values are
freed (ASan-clean) and absent from the rewritten log.
- **Given** delete+add of the same kind, **when** it boots, **then** refusal
naming both fields and the two-step alternative.
- **Given** a retype or a vanished class with rows, **when** it boots,
**then** refusal naming class, field and both shapes.
- **Given** pure declaration reordering, **when** it boots, **then** no
migration and every row in its right class *(closes the silent
cid-renumbering hole)*.
- **Given** a keys-resident table across a migration, **when** read,
**then** folds resolve through new offsets; a delta on a deleted field is
gone.
- **Given** a legacy log (no schema record) with an unchanged shape,
**when** it boots, **then** today's behaviour byte-for-byte, and the next
compaction writes the record.
- **Given** a kill between new-log write and rename, **when** the next boot
runs, **then** it re-migrates from the intact old log.
All met. Tests in `runtime/test/test_wal.c` unless noted.
- ✅ added field zero-valued, head carries the new schema —
`test_migrate_add_field`, and end-to-end at the language level
(`migrating \`Note\`: +flag`, row reads `flag=0`).
- ✅ deleted field's values freed and absent — `test_migrate_delete_field`
under ASan.
- ✅ same-kind delete+add refuses naming the two-step alternative —
`test_schema_diff_verdicts` case 5; bites-only-with-records proven by
`test_migrate_poison_needs_records`.
- ✅ retype / vanished class refuse by name — verdict cases 6–7; end-to-end
the retype refusal names `val` and exits 2, and the previous binary still
boots the refused log untouched.
- ✅ pure reordering migrates cids only — `test_migrate_reorder_owned`, which
also proves the cid INSIDE a stored owned value is renumbered.
- ✅ keys-resident across a migration — `test_migrate_delta_splice`: a chain
with a delta on the deleted field folds to the surviving field's latest
value after replay.
- ✅ legacy log unchanged-shape byte-for-byte — schema unset changes nothing
(all 5700 prior assertions), `test_schema_compaction_adopts_legacy` proves
head adoption.
- ✅ kill between write and rename — `test_migrate_crash_before_rename`
plants a COMPLETE valid migrated temp beside the untouched original; the
next boot discards it and re-migrates.
## Out of scope

View file

@ -1305,6 +1305,65 @@ static void test_migrate_delete_field(void) {
wo_db_destroy(&db2);
}
/* CRASH BETWEEN WRITE AND RENAME: the sharpest point on the timeline — a
* COMPLETE, VALID migrated log sits beside the original as the temp, the
* rename never happened. The next boot must treat the temp as the nothing it
* is (its records were never authoritative) and re-migrate from the intact
* original. A garbage temp tests the unlink; a valid one tests the doctrine. */
static void test_migrate_crash_before_rename(void) {
char pa[128], pb[160], tmp[160];
snprintf(pa, sizeof pa, "%s/migcrash.wal", g_dir);
snprintf(pb, sizeof pb, "%s/migcrash-copy.wal", g_dir);
snprintf(tmp, sizeof tmp, "%s.compact", pa);
wo_schema_class oc[] = {SC("row", 0, mig_sf_nt)};
wo_schema oldsc = {1, oc, NULL};
wo_schema_class nc[] = {SC("row", 0, mig_sf_nte)};
wo_schema newsc = {1, nc, NULL};
{
wo_db db;
T_EQ(wo_db_init(&db, MIG_NT, 1, 0, 1), 0);
wo_wal w;
T_EQ(wo_wal_open(&w, pa, 0), 0);
db_row *r = wo_row_create_raw(&db, 0, 1);
r->slots[0] = 5;
r->slots[1] = (uint64_t)(uintptr_t)mig_text("keep");
T_EQ(wo_row_raw_commit(&db, 0, r), 0);
T_EQ(wo_wal_append_insert(&w, &db, 0, 1), 0);
T_EQ(wo_wal_commit(&w), 0);
wo_wal_close(&w);
wo_db_destroy(&db);
}
wo_mig_plan pl;
T_EQ(wo_schema_diff(&oldsc, &newsc, &pl), 0);
/* produce the "crashed" state: migrate a COPY, then plant its result as
the original's temp — exactly what a kill after fsync, before rename,
leaves on disk */
{
FILE *a = fopen(pa, "rb"), *b = fopen(pb, "wb");
T_CHECK(a && b);
int ch;
while ((ch = fgetc(a)) != EOF) fputc(ch, b);
fclose(a);
fclose(b);
wo_db dbn;
T_EQ(wo_db_init(&dbn, MIG_NTE, 1, 0, 1), 0);
T_EQ(wo_wal_migrate(pb, &dbn, &oldsc, &pl, &newsc, 0, NULL), 0);
wo_db_destroy(&dbn);
T_EQ(rename(pb, tmp), 0);
}
/* the next boot: re-migrates from the intact original, result correct */
wo_db db2;
T_EQ(wo_db_init(&db2, MIG_NTE, 1, 0, 1), 0);
T_EQ(wo_wal_migrate(pa, &db2, &oldsc, &pl, &newsc, 0, NULL), 0);
wo_mig_plan_free(&pl);
T_EQ(wo_wal_replay(pa, &db2), 1);
db_row *r = wo_row_ptr(&db2, 0, 1);
T_CHECK(r != NULL && r->slots[0] == 5 && r->slots[2] == 0);
db_text *t = (db_text *)(uintptr_t)r->slots[1];
T_CHECK(t != NULL && t->len == 4 && memcmp(t->bytes, "keep", 4) == 0);
wo_db_destroy(&db2);
}
/* POISON BITES ONLY WITH RECORDS: a retyped class with no stored rows never
* blocks the boot; the same retype WITH a row refuses and names the field */
static void test_migrate_poison_needs_records(void) {
@ -3362,6 +3421,7 @@ int main(void) {
test_migrate_delta_splice();
test_migrate_add_field();
test_migrate_delete_field();
test_migrate_crash_before_rename();
test_migrate_poison_needs_records();
test_migrate_corrupt_input();
test_schema_diff_verdicts();