diff --git a/database/src/CODE-LOGIC.md b/database/src/CODE-LOGIC.md index e35bbbb..6fb7985 100644 --- a/database/src/CODE-LOGIC.md +++ b/database/src/CODE-LOGIC.md @@ -319,8 +319,44 @@ the `db_text*` layout) before the fix, pinned by 2. *Replay is O(N²) in a row's delta-chain length* — `apply_delta` folds the pre-delta row, and `wo_row_remove` (called internally) folds the SAME offset again, so each replayed delta re-walks its whole chain. -3. *Compaction triggers on byte ratio only* — `wo_wal_should_compact` has no - per-row delta-count signal, so one hot row (a single popular SKU) can grow - a long personal chain without moving the aggregate ratio enough to fire a - checkpoint. The no-chain-cap design decision rests on compaction bounding - length; for this shape it does not. +3. *Compaction triggers on byte ratio only* — **closed by databasev2 11**: + the fold reports hop count and `row_apply_field_keys` writes a full-row + image (`WO_WAL_UPDATE`) past `WO_DELTA_MAX_HOPS` (16), so a hot row's + chain is bounded in the update path itself; the checkpoint no longer + carries that burden. `wo_wal_should_compact` also gained an absolute + garbage term (`WO_CKPT_ABS_BYTES`). + +## Schema migrations (databasev2 12) + +A `@table` class is the schema; the log is the database; boot compares them. + +- **The log describes itself.** `WO_WAL_SCHEMA` (kind 5) is the head record + of every fresh and every compacted log: per class its NAME, storage flags, + and per field name + kind + the two encoding-relevant metadata words. + Written lazily by `stage()` ahead of the FIRST real record — never for a + log that stays empty, because `durable: false` programs have a documented + zero-bytes contract. `apply_record` skips it before reading cid/id (its + class count would be misread as a cid); replay does not count it. +- **The diff is name-keyed** (`wo_schema_diff`). Classes match by name, + fields by name + kind, owned references (`fclass`) by the NAME the number + resolves to — so pure declaration reordering costs only a cid remap, which + closes the old silent hole where reordering decoded rows into the wrong + class. Verdicts are per-class POISONS carried in the plan: retype, + same-shape delete+add (a disguised rename), vanished class, storage-flag + change, and the embed closure (any class whose stored values carry a + CHANGED class's old sub-shape, to a fixpoint). A poison forces the + transcode and bites only when a record of the class is actually met — no + rows, no verdict. +- **The migration is a record-level transcode** (`wo_wal_migrate`), not a + replay: no id maps, no indexes, no keys-resident logic. Old shapes decode + through a classdesc shim built from the stored schema; embedded cids are + renumbered by `mig_fixup_cids` (owned values carry a cid on the wire); + surviving fields move slots, deleted values are freed, added fields take + `enc_val(0)` — the kind's zero. Delta back-pointers rewrite through an + offset map, and a delta on a deleted field is SPLICED: it maps to its own + target, so later deltas step over it. Temp + fsync + rename, compaction's + own crash discipline — a kill anywhere leaves the old log authoritative, + including a kill after the temp is complete (`test_migrate_crash_before_rename`). +- **Legacy logs** (no head record) replay exactly as before and adopt the + head at their next compaction. v1 verbs are add and delete only; rename + wants `@renamed_from` (v2), data/seed migrations are v2. diff --git a/docs/guides/deploying-site.md b/docs/guides/deploying-site.md index f84212c..963a863 100644 --- a/docs/guides/deploying-site.md +++ b/docs/guides/deploying-site.md @@ -205,9 +205,14 @@ rm -rf /srv/writeonce-site/data && mv /srv/writeonce-site/data.bak- /srv/w systemctl start writeonce-site ``` -An older binary against a newer WAL is fine here only because the `Chapter` -schema has not changed. Once it does, that assumption dies and this section -needs a real answer. +Schema changes are handled since databasev2 12: a binary whose `@table` +classes gained or lost fields migrates the WAL at startup (the log's head +record carries the shape that wrote it), and an incompatible change — a +retyped field, a vanished class — refuses to start by name instead of +reporting corruption. Rolling BACK across a migration is itself a schema +change in the other direction: the old binary predates the head record's +shape, so expect the same refusal — restore the `.bak` data directory +alongside the old binary rather than pointing it at migrated data. ## Known gaps diff --git a/docs/stories/00-status.md b/docs/stories/00-status.md index 3a2e0cb..0f63af7 100644 --- a/docs/stories/00-status.md +++ b/docs/stories/00-status.md @@ -996,7 +996,7 @@ the language arc as v1 history. | 9 | [Cross-program tables](databasev2/09-cross-program-tables.md) *(was 20)* | ⏸ hold — attach to a running program's database over local IPC | | 10 | [Keypair attach auth](databasev2/10-keypair-attach-auth.md) *(was 21)* | ⏸ hold — program identity as a keypair; needs 9 | | 11 | [Bounded delta chains](databasev2/11-bounded-delta-chains.md) | ✅ **LANDED 2026-08-30.** A `resident: keys` row's delta chain is bounded in the UPDATE path, because the checkpoint is blind to per-row chain length — it thresholds on whole-log bytes, so one hot row can grow an unbounded chain inside a log that never trips compaction. The fold now reports hop count (free — the walk already visited every hop), and past `WO_DELTA_MAX_HOPS` (16) the update writes a full row image instead of a delta, resetting depth to 0. **Two things the tests corrected.** The flattened image is a `WO_WAL_UPDATE`, not an `INSERT`: the row's original INSERT is already in a live log, so a second one for the same id is a duplicate that replay correctly refuses as corruption — INSERT is right only for compaction, which builds a *fresh* log. And the **proportional ceiling was removed as dead code**: with the absolute term at 64 MiB, garbage large enough to reach a 256 MiB ceiling has already tripped it, so the branch was unreachable. Borrowing both constants from postgres was the wrong inference — PG needs two because it thresholds on *tuples* with its pair at opposite ends (base 50, max 1e8); this thresholds on *bytes*, where one constant does both jobs. Found by trying to write a test for the ceiling and finding no input could reach it. Four tests: depth stays bounded across 2K+2 updates, a flattened chain replays, a delta on an **indexed** column composes with flattening (checked at every step across the bound and after restart — found no product defect), and the policy's absolute term with its boundary. `test_wal` **5700 pass / 0 fail**; `wovm-test` and `woc-test` green. **One criterion is weaker than written:** the replay check asserts an expected value, not a `resident: all` oracle table. [spec](../superpowers/specs/2026-08-30-bounded-delta-chains-design.md) | -| 12 | [Schema migrations](databasev2/12-schema-migrations.md) | 🔄 **in progress (brainstormed + spec'd 2026-08-31).** A `@table` class is the schema, the log is the database, and today nothing compares them: an added or deleted field turns a healthy `WO_DATA` into "corruption" at boot, and reordering declarations silently decodes rows into the wrong class (records name classes by declaration index). v1, locked: **declarative and automatic at boot** — a `WO_WAL_SCHEMA` record at the log head states the shape; boot diffs it against the compiled classes by NAME; add and delete migrate through a compaction-style rewrite (added fields zero-filled — the grammar has no default syntax; deleted values freed at the rename swap); retype, same-kind delete+add (a disguised rename), and vanished classes REFUSE by name. Data/seed migrations deferred to v2. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) | +| 12 | [Schema migrations](databasev2/12-schema-migrations.md) | ✅ **LANDED 2026-08-31.** A `@table` class is the schema, the log is the database, and boot now compares them — before this, an added or deleted field turned a healthy `WO_DATA` into "corruption" and reordering declarations silently decoded rows into the wrong class. Landed: `WO_WAL_SCHEMA` head record (written LAZILY ahead of the first real record — an eager head broke `durable: false`'s documented zero-bytes contract by 75 bytes and the gate caught it), a name-keyed diff whose refusals are per-class POISONS that bite only when a record of the class is met, and a record-level TRANSCODE: cids remap by name including inside stored owned values, deleted values freed, added fields zero-filled, delta back-pointers rewritten through an offset map with deltas on deleted fields SPLICED out; temp+fsync+rename, compaction's crash discipline. **Two bugs the tests forced out:** a poisoned class skipped plan identity so the retype refusal fell through to generic "corruption" (the message this iteration exists to replace), and early `goto corrupt` freed uninitialized memory. End-to-end: `migrating \`Note\`: +flag` then `flag=0`; retype refuses naming `val`, exit 2, old binary still boots the refused log. 21 new tests, `test_wal` **5966/0**; wovm/woc/site/residency gates green. v2 holds rename (`@renamed_from`), retypes, and data/seed migrations. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) | --- diff --git a/docs/stories/databasev2/12-schema-migrations.md b/docs/stories/databasev2/12-schema-migrations.md index 0940b61..5800741 100644 --- a/docs/stories/databasev2/12-schema-migrations.md +++ b/docs/stories/databasev2/12-schema-migrations.md @@ -1,7 +1,7 @@ --- track: databasev2 iteration: "12" -status: in-progress +status: done readiness: ready --- @@ -36,35 +36,55 @@ readiness: ready | # | Task | State | | --- | --- | --- | -| 1 | `WO_WAL_SCHEMA` record: encode/decode, emitted by fresh-log open and by compaction | ⬜ | -| 2 | boot diff: peek head, match by class/field NAME, classify migrate / refuse / legacy | ⬜ | -| 3 | migration pass: old-shape replay through a field map, compaction-style rewrite, rename swap | ⬜ | -| 4 | refusals: retype, same-kind delete+add, vanished class — house-style messages | ⬜ | -| 5 | tests: the acceptance list, incl. keys-resident, legacy log, crash injection | ⬜ | -| 6 | close out: story, status board, deploy-guide note | ⬜ | +| 1 | `WO_WAL_SCHEMA` record: encode/decode, written LAZILY ahead of the first record and by compaction | ✅ `ba8519f` | +| 2 | boot diff: peek head, match by class/field NAME, classify migrate / refuse / legacy | ✅ `63a063b`, wired `b21943a` | +| 3 | migration pass: a record-level TRANSCODE (no replay, no db state), rename swap | ✅ `b69092a` | +| 4 | refusals: per-class poisons that bite only when a record of the class is met | ✅ `63a063b` + `b21943a` | +| 5 | tests: 21 new across three tiers, incl. delta splice, owned cid fixup, valid-temp crash | ✅ `test_wal` 5966/0 | +| 6 | close out: story, status board, deploy-guide note, CODE-LOGIC | ✅ this commit | + +## What changed against the plan, and why + +- **The migration pass became a record-level transcode**, not an old-shape + replay: the log is rewritten record by record (cids remapped by name, + fields moved/dropped/zero-filled, delta back-pointers rewritten through an + offset map) and the result replays through machinery that already exists + and is already tested. No id maps, no indexes, no keys-resident logic in + the migration itself. +- **The head record is written lazily**, ahead of the first real record. The + eager version broke `durable: false`'s documented zero-bytes contract by + 75 bytes; `residency-accept` caught it. +- **A poison forces the transcode instead of being skipped.** The first + end-to-end retype fell through to replay's generic "corruption" because a + poisoned class did not break plan identity — the exact message this + iteration exists to replace. Caught by the language-level smoke test. ## Acceptance criteria -- **Given** a log written with shape A and a binary with one added field, - **when** it boots, **then** it runs, the field reads zero-valued, and the - log head carries the new schema. -- **Given** one deleted field, **when** it boots, **then** dropped values are - freed (ASan-clean) and absent from the rewritten log. -- **Given** delete+add of the same kind, **when** it boots, **then** refusal - naming both fields and the two-step alternative. -- **Given** a retype or a vanished class with rows, **when** it boots, - **then** refusal naming class, field and both shapes. -- **Given** pure declaration reordering, **when** it boots, **then** no - migration and every row in its right class *(closes the silent - cid-renumbering hole)*. -- **Given** a keys-resident table across a migration, **when** read, - **then** folds resolve through new offsets; a delta on a deleted field is - gone. -- **Given** a legacy log (no schema record) with an unchanged shape, - **when** it boots, **then** today's behaviour byte-for-byte, and the next - compaction writes the record. -- **Given** a kill between new-log write and rename, **when** the next boot - runs, **then** it re-migrates from the intact old log. +All met. Tests in `runtime/test/test_wal.c` unless noted. + +- ✅ added field zero-valued, head carries the new schema — + `test_migrate_add_field`, and end-to-end at the language level + (`migrating \`Note\`: +flag`, row reads `flag=0`). +- ✅ deleted field's values freed and absent — `test_migrate_delete_field` + under ASan. +- ✅ same-kind delete+add refuses naming the two-step alternative — + `test_schema_diff_verdicts` case 5; bites-only-with-records proven by + `test_migrate_poison_needs_records`. +- ✅ retype / vanished class refuse by name — verdict cases 6–7; end-to-end + the retype refusal names `val` and exits 2, and the previous binary still + boots the refused log untouched. +- ✅ pure reordering migrates cids only — `test_migrate_reorder_owned`, which + also proves the cid INSIDE a stored owned value is renumbered. +- ✅ keys-resident across a migration — `test_migrate_delta_splice`: a chain + with a delta on the deleted field folds to the surviving field's latest + value after replay. +- ✅ legacy log unchanged-shape byte-for-byte — schema unset changes nothing + (all 5700 prior assertions), `test_schema_compaction_adopts_legacy` proves + head adoption. +- ✅ kill between write and rename — `test_migrate_crash_before_rename` + plants a COMPLETE valid migrated temp beside the untouched original; the + next boot discards it and re-migrates. ## Out of scope diff --git a/runtime/test/test_wal.c b/runtime/test/test_wal.c index 5cac5b8..cb61554 100644 --- a/runtime/test/test_wal.c +++ b/runtime/test/test_wal.c @@ -1305,6 +1305,65 @@ static void test_migrate_delete_field(void) { wo_db_destroy(&db2); } +/* CRASH BETWEEN WRITE AND RENAME: the sharpest point on the timeline — a + * COMPLETE, VALID migrated log sits beside the original as the temp, the + * rename never happened. The next boot must treat the temp as the nothing it + * is (its records were never authoritative) and re-migrate from the intact + * original. A garbage temp tests the unlink; a valid one tests the doctrine. */ +static void test_migrate_crash_before_rename(void) { + char pa[128], pb[160], tmp[160]; + snprintf(pa, sizeof pa, "%s/migcrash.wal", g_dir); + snprintf(pb, sizeof pb, "%s/migcrash-copy.wal", g_dir); + snprintf(tmp, sizeof tmp, "%s.compact", pa); + wo_schema_class oc[] = {SC("row", 0, mig_sf_nt)}; + wo_schema oldsc = {1, oc, NULL}; + wo_schema_class nc[] = {SC("row", 0, mig_sf_nte)}; + wo_schema newsc = {1, nc, NULL}; + { + wo_db db; + T_EQ(wo_db_init(&db, MIG_NT, 1, 0, 1), 0); + wo_wal w; + T_EQ(wo_wal_open(&w, pa, 0), 0); + db_row *r = wo_row_create_raw(&db, 0, 1); + r->slots[0] = 5; + r->slots[1] = (uint64_t)(uintptr_t)mig_text("keep"); + T_EQ(wo_row_raw_commit(&db, 0, r), 0); + T_EQ(wo_wal_append_insert(&w, &db, 0, 1), 0); + T_EQ(wo_wal_commit(&w), 0); + wo_wal_close(&w); + wo_db_destroy(&db); + } + wo_mig_plan pl; + T_EQ(wo_schema_diff(&oldsc, &newsc, &pl), 0); + /* produce the "crashed" state: migrate a COPY, then plant its result as + the original's temp — exactly what a kill after fsync, before rename, + leaves on disk */ + { + FILE *a = fopen(pa, "rb"), *b = fopen(pb, "wb"); + T_CHECK(a && b); + int ch; + while ((ch = fgetc(a)) != EOF) fputc(ch, b); + fclose(a); + fclose(b); + wo_db dbn; + T_EQ(wo_db_init(&dbn, MIG_NTE, 1, 0, 1), 0); + T_EQ(wo_wal_migrate(pb, &dbn, &oldsc, &pl, &newsc, 0, NULL), 0); + wo_db_destroy(&dbn); + T_EQ(rename(pb, tmp), 0); + } + /* the next boot: re-migrates from the intact original, result correct */ + wo_db db2; + T_EQ(wo_db_init(&db2, MIG_NTE, 1, 0, 1), 0); + T_EQ(wo_wal_migrate(pa, &db2, &oldsc, &pl, &newsc, 0, NULL), 0); + wo_mig_plan_free(&pl); + T_EQ(wo_wal_replay(pa, &db2), 1); + db_row *r = wo_row_ptr(&db2, 0, 1); + T_CHECK(r != NULL && r->slots[0] == 5 && r->slots[2] == 0); + db_text *t = (db_text *)(uintptr_t)r->slots[1]; + T_CHECK(t != NULL && t->len == 4 && memcmp(t->bytes, "keep", 4) == 0); + wo_db_destroy(&db2); +} + /* POISON BITES ONLY WITH RECORDS: a retyped class with no stored rows never * blocks the boot; the same retype WITH a row refuses and names the field */ static void test_migrate_poison_needs_records(void) { @@ -3362,6 +3421,7 @@ int main(void) { test_migrate_delta_splice(); test_migrate_add_field(); test_migrate_delete_field(); + test_migrate_crash_before_rename(); test_migrate_poison_needs_records(); test_migrate_corrupt_input(); test_schema_diff_verdicts();