docs(db2-migrate): close out iteration 12
- crash-before-rename test: a COMPLETE valid migrated temp beside the untouched original is discarded and the boot re-migrates — the sharpest point on the crash timeline, deterministic, no fault injection needed - story: all six tasks done with commit hashes, all eight criteria met with the test that proves each, plus the three deviations from the plan and why (transcode over replay, lazy head, poison forces transcode) - CODE-LOGIC: migration section; also corrected limitation 3, which still claimed unbounded hot-row chains — iteration 11 closed that - status board row 12; deploy guide's rollback section gets its real answer (rolling back across a migration is a migration backwards: expect the refusal, restore the .bak) - test_wal 5966 pass, 0 fail Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 4bb6ece2531e2123eb958c91d9bef4a6528eab3b)
This commit is contained in:
parent
27aecd3dc1
commit
4ad24d6381
5 changed files with 157 additions and 36 deletions
|
|
@ -319,8 +319,44 @@ the `db_text*` layout) before the fix, pinned by
|
|||
2. *Replay is O(N²) in a row's delta-chain length* — `apply_delta` folds the
|
||||
pre-delta row, and `wo_row_remove` (called internally) folds the SAME
|
||||
offset again, so each replayed delta re-walks its whole chain.
|
||||
3. *Compaction triggers on byte ratio only* — `wo_wal_should_compact` has no
|
||||
per-row delta-count signal, so one hot row (a single popular SKU) can grow
|
||||
a long personal chain without moving the aggregate ratio enough to fire a
|
||||
checkpoint. The no-chain-cap design decision rests on compaction bounding
|
||||
length; for this shape it does not.
|
||||
3. *Compaction triggers on byte ratio only* — **closed by databasev2 11**:
|
||||
the fold reports hop count and `row_apply_field_keys` writes a full-row
|
||||
image (`WO_WAL_UPDATE`) past `WO_DELTA_MAX_HOPS` (16), so a hot row's
|
||||
chain is bounded in the update path itself; the checkpoint no longer
|
||||
carries that burden. `wo_wal_should_compact` also gained an absolute
|
||||
garbage term (`WO_CKPT_ABS_BYTES`).
|
||||
|
||||
## Schema migrations (databasev2 12)
|
||||
|
||||
A `@table` class is the schema; the log is the database; boot compares them.
|
||||
|
||||
- **The log describes itself.** `WO_WAL_SCHEMA` (kind 5) is the head record
|
||||
of every fresh and every compacted log: per class its NAME, storage flags,
|
||||
and per field name + kind + the two encoding-relevant metadata words.
|
||||
Written lazily by `stage()` ahead of the FIRST real record — never for a
|
||||
log that stays empty, because `durable: false` programs have a documented
|
||||
zero-bytes contract. `apply_record` skips it before reading cid/id (its
|
||||
class count would be misread as a cid); replay does not count it.
|
||||
- **The diff is name-keyed** (`wo_schema_diff`). Classes match by name,
|
||||
fields by name + kind, owned references (`fclass`) by the NAME the number
|
||||
resolves to — so pure declaration reordering costs only a cid remap, which
|
||||
closes the old silent hole where reordering decoded rows into the wrong
|
||||
class. Verdicts are per-class POISONS carried in the plan: retype,
|
||||
same-shape delete+add (a disguised rename), vanished class, storage-flag
|
||||
change, and the embed closure (any class whose stored values carry a
|
||||
CHANGED class's old sub-shape, to a fixpoint). A poison forces the
|
||||
transcode and bites only when a record of the class is actually met — no
|
||||
rows, no verdict.
|
||||
- **The migration is a record-level transcode** (`wo_wal_migrate`), not a
|
||||
replay: no id maps, no indexes, no keys-resident logic. Old shapes decode
|
||||
through a classdesc shim built from the stored schema; embedded cids are
|
||||
renumbered by `mig_fixup_cids` (owned values carry a cid on the wire);
|
||||
surviving fields move slots, deleted values are freed, added fields take
|
||||
`enc_val(0)` — the kind's zero. Delta back-pointers rewrite through an
|
||||
offset map, and a delta on a deleted field is SPLICED: it maps to its own
|
||||
target, so later deltas step over it. Temp + fsync + rename, compaction's
|
||||
own crash discipline — a kill anywhere leaves the old log authoritative,
|
||||
including a kill after the temp is complete (`test_migrate_crash_before_rename`).
|
||||
- **Legacy logs** (no head record) replay exactly as before and adopt the
|
||||
head at their next compaction. v1 verbs are add and delete only; rename
|
||||
wants `@renamed_from` (v2), data/seed migrations are v2.
|
||||
|
|
|
|||
|
|
@ -205,9 +205,14 @@ rm -rf /srv/writeonce-site/data && mv /srv/writeonce-site/data.bak-<date> /srv/w
|
|||
systemctl start writeonce-site
|
||||
```
|
||||
|
||||
An older binary against a newer WAL is fine here only because the `Chapter`
|
||||
schema has not changed. Once it does, that assumption dies and this section
|
||||
needs a real answer.
|
||||
Schema changes are handled since databasev2 12: a binary whose `@table`
|
||||
classes gained or lost fields migrates the WAL at startup (the log's head
|
||||
record carries the shape that wrote it), and an incompatible change — a
|
||||
retyped field, a vanished class — refuses to start by name instead of
|
||||
reporting corruption. Rolling BACK across a migration is itself a schema
|
||||
change in the other direction: the old binary predates the head record's
|
||||
shape, so expect the same refusal — restore the `.bak` data directory
|
||||
alongside the old binary rather than pointing it at migrated data.
|
||||
|
||||
## Known gaps
|
||||
|
||||
|
|
|
|||
|
|
@ -996,7 +996,7 @@ the language arc as v1 history.
|
|||
| 9 | [Cross-program tables](databasev2/09-cross-program-tables.md) *(was 20)* | ⏸ hold — attach to a running program's database over local IPC |
|
||||
| 10 | [Keypair attach auth](databasev2/10-keypair-attach-auth.md) *(was 21)* | ⏸ hold — program identity as a keypair; needs 9 |
|
||||
| 11 | [Bounded delta chains](databasev2/11-bounded-delta-chains.md) | ✅ **LANDED 2026-08-30.** A `resident: keys` row's delta chain is bounded in the UPDATE path, because the checkpoint is blind to per-row chain length — it thresholds on whole-log bytes, so one hot row can grow an unbounded chain inside a log that never trips compaction. The fold now reports hop count (free — the walk already visited every hop), and past `WO_DELTA_MAX_HOPS` (16) the update writes a full row image instead of a delta, resetting depth to 0. **Two things the tests corrected.** The flattened image is a `WO_WAL_UPDATE`, not an `INSERT`: the row's original INSERT is already in a live log, so a second one for the same id is a duplicate that replay correctly refuses as corruption — INSERT is right only for compaction, which builds a *fresh* log. And the **proportional ceiling was removed as dead code**: with the absolute term at 64 MiB, garbage large enough to reach a 256 MiB ceiling has already tripped it, so the branch was unreachable. Borrowing both constants from postgres was the wrong inference — PG needs two because it thresholds on *tuples* with its pair at opposite ends (base 50, max 1e8); this thresholds on *bytes*, where one constant does both jobs. Found by trying to write a test for the ceiling and finding no input could reach it. Four tests: depth stays bounded across 2K+2 updates, a flattened chain replays, a delta on an **indexed** column composes with flattening (checked at every step across the bound and after restart — found no product defect), and the policy's absolute term with its boundary. `test_wal` **5700 pass / 0 fail**; `wovm-test` and `woc-test` green. **One criterion is weaker than written:** the replay check asserts an expected value, not a `resident: all` oracle table. [spec](../superpowers/specs/2026-08-30-bounded-delta-chains-design.md) |
|
||||
| 12 | [Schema migrations](databasev2/12-schema-migrations.md) | 🔄 **in progress (brainstormed + spec'd 2026-08-31).** A `@table` class is the schema, the log is the database, and today nothing compares them: an added or deleted field turns a healthy `WO_DATA` into "corruption" at boot, and reordering declarations silently decodes rows into the wrong class (records name classes by declaration index). v1, locked: **declarative and automatic at boot** — a `WO_WAL_SCHEMA` record at the log head states the shape; boot diffs it against the compiled classes by NAME; add and delete migrate through a compaction-style rewrite (added fields zero-filled — the grammar has no default syntax; deleted values freed at the rename swap); retype, same-kind delete+add (a disguised rename), and vanished classes REFUSE by name. Data/seed migrations deferred to v2. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) |
|
||||
| 12 | [Schema migrations](databasev2/12-schema-migrations.md) | ✅ **LANDED 2026-08-31.** A `@table` class is the schema, the log is the database, and boot now compares them — before this, an added or deleted field turned a healthy `WO_DATA` into "corruption" and reordering declarations silently decoded rows into the wrong class. Landed: `WO_WAL_SCHEMA` head record (written LAZILY ahead of the first real record — an eager head broke `durable: false`'s documented zero-bytes contract by 75 bytes and the gate caught it), a name-keyed diff whose refusals are per-class POISONS that bite only when a record of the class is met, and a record-level TRANSCODE: cids remap by name including inside stored owned values, deleted values freed, added fields zero-filled, delta back-pointers rewritten through an offset map with deltas on deleted fields SPLICED out; temp+fsync+rename, compaction's crash discipline. **Two bugs the tests forced out:** a poisoned class skipped plan identity so the retype refusal fell through to generic "corruption" (the message this iteration exists to replace), and early `goto corrupt` freed uninitialized memory. End-to-end: `migrating \`Note\`: +flag` then `flag=0`; retype refuses naming `val`, exit 2, old binary still boots the refused log. 21 new tests, `test_wal` **5966/0**; wovm/woc/site/residency gates green. v2 holds rename (`@renamed_from`), retypes, and data/seed migrations. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
track: databasev2
|
||||
iteration: "12"
|
||||
status: in-progress
|
||||
status: done
|
||||
readiness: ready
|
||||
---
|
||||
|
||||
|
|
@ -36,35 +36,55 @@ readiness: ready
|
|||
|
||||
| # | Task | State |
|
||||
| --- | --- | --- |
|
||||
| 1 | `WO_WAL_SCHEMA` record: encode/decode, emitted by fresh-log open and by compaction | ⬜ |
|
||||
| 2 | boot diff: peek head, match by class/field NAME, classify migrate / refuse / legacy | ⬜ |
|
||||
| 3 | migration pass: old-shape replay through a field map, compaction-style rewrite, rename swap | ⬜ |
|
||||
| 4 | refusals: retype, same-kind delete+add, vanished class — house-style messages | ⬜ |
|
||||
| 5 | tests: the acceptance list, incl. keys-resident, legacy log, crash injection | ⬜ |
|
||||
| 6 | close out: story, status board, deploy-guide note | ⬜ |
|
||||
| 1 | `WO_WAL_SCHEMA` record: encode/decode, written LAZILY ahead of the first record and by compaction | ✅ `ba8519f` |
|
||||
| 2 | boot diff: peek head, match by class/field NAME, classify migrate / refuse / legacy | ✅ `63a063b`, wired `b21943a` |
|
||||
| 3 | migration pass: a record-level TRANSCODE (no replay, no db state), rename swap | ✅ `b69092a` |
|
||||
| 4 | refusals: per-class poisons that bite only when a record of the class is met | ✅ `63a063b` + `b21943a` |
|
||||
| 5 | tests: 21 new across three tiers, incl. delta splice, owned cid fixup, valid-temp crash | ✅ `test_wal` 5966/0 |
|
||||
| 6 | close out: story, status board, deploy-guide note, CODE-LOGIC | ✅ this commit |
|
||||
|
||||
## What changed against the plan, and why
|
||||
|
||||
- **The migration pass became a record-level transcode**, not an old-shape
|
||||
replay: the log is rewritten record by record (cids remapped by name,
|
||||
fields moved/dropped/zero-filled, delta back-pointers rewritten through an
|
||||
offset map) and the result replays through machinery that already exists
|
||||
and is already tested. No id maps, no indexes, no keys-resident logic in
|
||||
the migration itself.
|
||||
- **The head record is written lazily**, ahead of the first real record. The
|
||||
eager version broke `durable: false`'s documented zero-bytes contract by
|
||||
75 bytes; `residency-accept` caught it.
|
||||
- **A poison forces the transcode instead of being skipped.** The first
|
||||
end-to-end retype fell through to replay's generic "corruption" because a
|
||||
poisoned class did not break plan identity — the exact message this
|
||||
iteration exists to replace. Caught by the language-level smoke test.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **Given** a log written with shape A and a binary with one added field,
|
||||
**when** it boots, **then** it runs, the field reads zero-valued, and the
|
||||
log head carries the new schema.
|
||||
- **Given** one deleted field, **when** it boots, **then** dropped values are
|
||||
freed (ASan-clean) and absent from the rewritten log.
|
||||
- **Given** delete+add of the same kind, **when** it boots, **then** refusal
|
||||
naming both fields and the two-step alternative.
|
||||
- **Given** a retype or a vanished class with rows, **when** it boots,
|
||||
**then** refusal naming class, field and both shapes.
|
||||
- **Given** pure declaration reordering, **when** it boots, **then** no
|
||||
migration and every row in its right class *(closes the silent
|
||||
cid-renumbering hole)*.
|
||||
- **Given** a keys-resident table across a migration, **when** read,
|
||||
**then** folds resolve through new offsets; a delta on a deleted field is
|
||||
gone.
|
||||
- **Given** a legacy log (no schema record) with an unchanged shape,
|
||||
**when** it boots, **then** today's behaviour byte-for-byte, and the next
|
||||
compaction writes the record.
|
||||
- **Given** a kill between new-log write and rename, **when** the next boot
|
||||
runs, **then** it re-migrates from the intact old log.
|
||||
All met. Tests in `runtime/test/test_wal.c` unless noted.
|
||||
|
||||
- ✅ added field zero-valued, head carries the new schema —
|
||||
`test_migrate_add_field`, and end-to-end at the language level
|
||||
(`migrating \`Note\`: +flag`, row reads `flag=0`).
|
||||
- ✅ deleted field's values freed and absent — `test_migrate_delete_field`
|
||||
under ASan.
|
||||
- ✅ same-kind delete+add refuses naming the two-step alternative —
|
||||
`test_schema_diff_verdicts` case 5; bites-only-with-records proven by
|
||||
`test_migrate_poison_needs_records`.
|
||||
- ✅ retype / vanished class refuse by name — verdict cases 6–7; end-to-end
|
||||
the retype refusal names `val` and exits 2, and the previous binary still
|
||||
boots the refused log untouched.
|
||||
- ✅ pure reordering migrates cids only — `test_migrate_reorder_owned`, which
|
||||
also proves the cid INSIDE a stored owned value is renumbered.
|
||||
- ✅ keys-resident across a migration — `test_migrate_delta_splice`: a chain
|
||||
with a delta on the deleted field folds to the surviving field's latest
|
||||
value after replay.
|
||||
- ✅ legacy log unchanged-shape byte-for-byte — schema unset changes nothing
|
||||
(all 5700 prior assertions), `test_schema_compaction_adopts_legacy` proves
|
||||
head adoption.
|
||||
- ✅ kill between write and rename — `test_migrate_crash_before_rename`
|
||||
plants a COMPLETE valid migrated temp beside the untouched original; the
|
||||
next boot discards it and re-migrates.
|
||||
|
||||
## Out of scope
|
||||
|
||||
|
|
|
|||
|
|
@ -1305,6 +1305,65 @@ static void test_migrate_delete_field(void) {
|
|||
wo_db_destroy(&db2);
|
||||
}
|
||||
|
||||
/* CRASH BETWEEN WRITE AND RENAME: the sharpest point on the timeline — a
|
||||
* COMPLETE, VALID migrated log sits beside the original as the temp, the
|
||||
* rename never happened. The next boot must treat the temp as the nothing it
|
||||
* is (its records were never authoritative) and re-migrate from the intact
|
||||
* original. A garbage temp tests the unlink; a valid one tests the doctrine. */
|
||||
static void test_migrate_crash_before_rename(void) {
|
||||
char pa[128], pb[160], tmp[160];
|
||||
snprintf(pa, sizeof pa, "%s/migcrash.wal", g_dir);
|
||||
snprintf(pb, sizeof pb, "%s/migcrash-copy.wal", g_dir);
|
||||
snprintf(tmp, sizeof tmp, "%s.compact", pa);
|
||||
wo_schema_class oc[] = {SC("row", 0, mig_sf_nt)};
|
||||
wo_schema oldsc = {1, oc, NULL};
|
||||
wo_schema_class nc[] = {SC("row", 0, mig_sf_nte)};
|
||||
wo_schema newsc = {1, nc, NULL};
|
||||
{
|
||||
wo_db db;
|
||||
T_EQ(wo_db_init(&db, MIG_NT, 1, 0, 1), 0);
|
||||
wo_wal w;
|
||||
T_EQ(wo_wal_open(&w, pa, 0), 0);
|
||||
db_row *r = wo_row_create_raw(&db, 0, 1);
|
||||
r->slots[0] = 5;
|
||||
r->slots[1] = (uint64_t)(uintptr_t)mig_text("keep");
|
||||
T_EQ(wo_row_raw_commit(&db, 0, r), 0);
|
||||
T_EQ(wo_wal_append_insert(&w, &db, 0, 1), 0);
|
||||
T_EQ(wo_wal_commit(&w), 0);
|
||||
wo_wal_close(&w);
|
||||
wo_db_destroy(&db);
|
||||
}
|
||||
wo_mig_plan pl;
|
||||
T_EQ(wo_schema_diff(&oldsc, &newsc, &pl), 0);
|
||||
/* produce the "crashed" state: migrate a COPY, then plant its result as
|
||||
the original's temp — exactly what a kill after fsync, before rename,
|
||||
leaves on disk */
|
||||
{
|
||||
FILE *a = fopen(pa, "rb"), *b = fopen(pb, "wb");
|
||||
T_CHECK(a && b);
|
||||
int ch;
|
||||
while ((ch = fgetc(a)) != EOF) fputc(ch, b);
|
||||
fclose(a);
|
||||
fclose(b);
|
||||
wo_db dbn;
|
||||
T_EQ(wo_db_init(&dbn, MIG_NTE, 1, 0, 1), 0);
|
||||
T_EQ(wo_wal_migrate(pb, &dbn, &oldsc, &pl, &newsc, 0, NULL), 0);
|
||||
wo_db_destroy(&dbn);
|
||||
T_EQ(rename(pb, tmp), 0);
|
||||
}
|
||||
/* the next boot: re-migrates from the intact original, result correct */
|
||||
wo_db db2;
|
||||
T_EQ(wo_db_init(&db2, MIG_NTE, 1, 0, 1), 0);
|
||||
T_EQ(wo_wal_migrate(pa, &db2, &oldsc, &pl, &newsc, 0, NULL), 0);
|
||||
wo_mig_plan_free(&pl);
|
||||
T_EQ(wo_wal_replay(pa, &db2), 1);
|
||||
db_row *r = wo_row_ptr(&db2, 0, 1);
|
||||
T_CHECK(r != NULL && r->slots[0] == 5 && r->slots[2] == 0);
|
||||
db_text *t = (db_text *)(uintptr_t)r->slots[1];
|
||||
T_CHECK(t != NULL && t->len == 4 && memcmp(t->bytes, "keep", 4) == 0);
|
||||
wo_db_destroy(&db2);
|
||||
}
|
||||
|
||||
/* POISON BITES ONLY WITH RECORDS: a retyped class with no stored rows never
|
||||
* blocks the boot; the same retype WITH a row refuses and names the field */
|
||||
static void test_migrate_poison_needs_records(void) {
|
||||
|
|
@ -3362,6 +3421,7 @@ int main(void) {
|
|||
test_migrate_delta_splice();
|
||||
test_migrate_add_field();
|
||||
test_migrate_delete_field();
|
||||
test_migrate_crash_before_rename();
|
||||
test_migrate_poison_needs_records();
|
||||
test_migrate_corrupt_input();
|
||||
test_schema_diff_verdicts();
|
||||
|
|
|
|||
Loading…
Reference in a new issue