diff --git a/docs/00-git-commit-history.md b/docs/00-git-commit-history.md index bd8a53e..d3535df 100644 --- a/docs/00-git-commit-history.md +++ b/docs/00-git-commit-history.md @@ -37,6 +37,7 @@ features cannot collide. | `db2-delta` | databasev2 2 — keys-resident updates as WAL delta records | ✅ on `master` 2026-08-30 | | `db2-chains` / `db2-chain` | databasev2 11 — bounding a keys-resident row's delta chain | ✅ on `master` 2026-08-30 | | `site` | the writeonce.de tutorial site | ✅ on `master` 2026-08-30 | +| `db2-migrate` | databasev2 12 — schema migrations (add/delete, declarative, auto on boot) | claimed 2026-08-31, work on `dev` | | `site-submodule` | `docs/examples/site` extracted to github.com/shoneyJ/writeonce-site and consumed as a submodule | ✅ on `master` 2026-08-30. Both branches now track the site by revision; an edit to it is a commit in that repo plus a pointer bump here | | `lang41` | runtime: unadopted shard must not impersonate shard 0 | on `dev` (`9dca0b4`); independent of the residency stack, not picked | | `porch-store` | porch store tables, Limiter and Idempotent middleware (Phases A, B, C) | on `dev` (`519d411`, `5b1e82a`, `aee7926`). **In progress**: Phase C was uncommitted work from a parallel session, committed as-is, and calls `json.decode`/`json.encode` with no `use json` import | diff --git a/docs/stories/00-status.md b/docs/stories/00-status.md index c2d7671..3a2e0cb 100644 --- a/docs/stories/00-status.md +++ b/docs/stories/00-status.md @@ -996,6 +996,7 @@ the language arc as v1 history. | 9 | [Cross-program tables](databasev2/09-cross-program-tables.md) *(was 20)* | ⏸ hold — attach to a running program's database over local IPC | | 10 | [Keypair attach auth](databasev2/10-keypair-attach-auth.md) *(was 21)* | ⏸ hold — program identity as a keypair; needs 9 | | 11 | [Bounded delta chains](databasev2/11-bounded-delta-chains.md) | ✅ **LANDED 2026-08-30.** A `resident: keys` row's delta chain is bounded in the UPDATE path, because the checkpoint is blind to per-row chain length — it thresholds on whole-log bytes, so one hot row can grow an unbounded chain inside a log that never trips compaction. The fold now reports hop count (free — the walk already visited every hop), and past `WO_DELTA_MAX_HOPS` (16) the update writes a full row image instead of a delta, resetting depth to 0. **Two things the tests corrected.** The flattened image is a `WO_WAL_UPDATE`, not an `INSERT`: the row's original INSERT is already in a live log, so a second one for the same id is a duplicate that replay correctly refuses as corruption — INSERT is right only for compaction, which builds a *fresh* log. And the **proportional ceiling was removed as dead code**: with the absolute term at 64 MiB, garbage large enough to reach a 256 MiB ceiling has already tripped it, so the branch was unreachable. Borrowing both constants from postgres was the wrong inference — PG needs two because it thresholds on *tuples* with its pair at opposite ends (base 50, max 1e8); this thresholds on *bytes*, where one constant does both jobs. Found by trying to write a test for the ceiling and finding no input could reach it. Four tests: depth stays bounded across 2K+2 updates, a flattened chain replays, a delta on an **indexed** column composes with flattening (checked at every step across the bound and after restart — found no product defect), and the policy's absolute term with its boundary. `test_wal` **5700 pass / 0 fail**; `wovm-test` and `woc-test` green. **One criterion is weaker than written:** the replay check asserts an expected value, not a `resident: all` oracle table. [spec](../superpowers/specs/2026-08-30-bounded-delta-chains-design.md) | +| 12 | [Schema migrations](databasev2/12-schema-migrations.md) | 🔄 **in progress (brainstormed + spec'd 2026-08-31).** A `@table` class is the schema, the log is the database, and today nothing compares them: an added or deleted field turns a healthy `WO_DATA` into "corruption" at boot, and reordering declarations silently decodes rows into the wrong class (records name classes by declaration index). v1, locked: **declarative and automatic at boot** — a `WO_WAL_SCHEMA` record at the log head states the shape; boot diffs it against the compiled classes by NAME; add and delete migrate through a compaction-style rewrite (added fields zero-filled — the grammar has no default syntax; deleted values freed at the rename swap); retype, same-kind delete+add (a disguised rename), and vanished classes REFUSE by name. Data/seed migrations deferred to v2. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) | --- diff --git a/docs/stories/databasev2/12-schema-migrations.md b/docs/stories/databasev2/12-schema-migrations.md new file mode 100644 index 0000000..0940b61 --- /dev/null +++ b/docs/stories/databasev2/12-schema-migrations.md @@ -0,0 +1,77 @@ +--- +track: databasev2 +iteration: "12" +status: in-progress +readiness: ready +--- + +# databasev2 12 — schema migrations: add + delete, declarative, auto on boot + +> Part of [Story — databasev2: the database beyond RAM](00-story.md). +> Spec: [`2026-08-31-schema-migrations-design.md`](../../superpowers/specs/2026-08-31-schema-migrations-design.md). +> +> A `@table` class is the schema and the log is the database, but nothing +> compares them: today an added or deleted field turns a healthy `WO_DATA` +> into "corruption" at boot, and reordering declarations decodes rows into +> the wrong class without a sound. Changing the class IS a migration — this +> iteration makes the runtime perform the two unambiguous ones and refuse +> the rest by name. + +## Decisions (locked in brainstorm, 2026-08-31) + +- **Declarative, automatic, at boot** — the binary carries the schema; no + migration files, no offline tool. Chosen over migration-code-in-language + and an offline `--migrate` step. +- **v1 verbs: add + delete only.** Added fields take the kind's zero value — + the grammar has no field-default syntax and v1 does not grow compiler + surface. Deleted fields' values are freed and physically dropped at the + rename swap. +- **Same-kind delete+add in one step refuses** — indistinguishable from a + rename, and the wrong guess destroys data. Two-step deploy or v2's + `@renamed_from`. +- **Data/seed migrations are v2** — the deploy guide's trap 1 (seed only + fills an empty table) stays documented, not fixed. + +## Tasks + +| # | Task | State | +| --- | --- | --- | +| 1 | `WO_WAL_SCHEMA` record: encode/decode, emitted by fresh-log open and by compaction | ⬜ | +| 2 | boot diff: peek head, match by class/field NAME, classify migrate / refuse / legacy | ⬜ | +| 3 | migration pass: old-shape replay through a field map, compaction-style rewrite, rename swap | ⬜ | +| 4 | refusals: retype, same-kind delete+add, vanished class — house-style messages | ⬜ | +| 5 | tests: the acceptance list, incl. keys-resident, legacy log, crash injection | ⬜ | +| 6 | close out: story, status board, deploy-guide note | ⬜ | + +## Acceptance criteria + +- **Given** a log written with shape A and a binary with one added field, + **when** it boots, **then** it runs, the field reads zero-valued, and the + log head carries the new schema. +- **Given** one deleted field, **when** it boots, **then** dropped values are + freed (ASan-clean) and absent from the rewritten log. +- **Given** delete+add of the same kind, **when** it boots, **then** refusal + naming both fields and the two-step alternative. +- **Given** a retype or a vanished class with rows, **when** it boots, + **then** refusal naming class, field and both shapes. +- **Given** pure declaration reordering, **when** it boots, **then** no + migration and every row in its right class *(closes the silent + cid-renumbering hole)*. +- **Given** a keys-resident table across a migration, **when** read, + **then** folds resolve through new offsets; a delta on a deleted field is + gone. +- **Given** a legacy log (no schema record) with an unchanged shape, + **when** it boots, **then** today's behaviour byte-for-byte, and the next + compaction writes the record. +- **Given** a kill between new-log write and rename, **when** the next boot + runs, **then** it re-migrates from the intact old log. + +## Out of scope + +- **Rename** — v2, via `@renamed_from`; v1 refuses the ambiguous diff. +- **Retype/conversions** — refusal costs one explicit backfill program. +- **Dropped-class verdicts** — refusing keeps the data; deletion is a + decision, not a migration. +- **`@seed(n)` data migrations** — deferred by decision. +- **Field-default syntax** — zero-fill plus app code covers it. +- **Index migrations** — indexes rebuild at boot; refusal is about flags. diff --git a/docs/superpowers/specs/2026-08-31-schema-migrations-design.md b/docs/superpowers/specs/2026-08-31-schema-migrations-design.md new file mode 100644 index 0000000..f521710 --- /dev/null +++ b/docs/superpowers/specs/2026-08-31-schema-migrations-design.md @@ -0,0 +1,135 @@ +# Schema migrations v1 — add + delete, declarative, auto on boot + +> Brainstormed and approved 2026-08-31. Story: +> [databasev2 12](../../stories/databasev2/12-schema-migrations.md). +> Scope settled during brainstorm: declarative model, add/delete only, +> zero-value fill, data/seed migrations deferred to v2. + +## The problem + +A `@table` class is the schema and `WO_DATA`'s log is the database, but the +two are never compared. A record names its class by `cid` — the class's +declaration index — and replay decodes every record against the *compiled* +shape. So today, all verified: + +- Adding a field makes every old record under-run its decode and replay + refuses the whole log as "corruption". +- Deleting a field leaves trailing bytes — the same refusal. +- Reordering `@table` declarations renumbers cids and rows decode into the + wrong class, silently when shapes happen to match. +- Nothing records which schema wrote the log. The durable/resident *mode* + mismatch is checked at boot (databasev2 2); the shape is not. + +Changing a `@table` class is a database migration. The language performs it. + +## The model, and why + +**Declarative, automatic, at boot.** The binary already carries the schema — +`wo_classdesc` has the class name and per-field names (v2 metadata), so no +`.wob` format change is needed. At startup the runtime compares the shape +recorded in the log against the compiled shape and either replays (match), +migrates (add/delete), or refuses (everything else). No migration files, no +offline step: writeonce's doctrine is one binary and no ops choreography, and +a migration artifact would be a second source of truth for a schema the class +declaration already states. + +**v1 verbs: add and delete only.** They are the two changes whose data +consequence is unambiguous: an added field has no stored values and takes the +kind's zero value (Int 0, Float 0.0, Bool false, Text empty, optional nil); +a deleted field's stored values are freed and physically dropped when the +rewritten log is swapped in. Zero-fill is chosen over a default-value syntax +because the grammar has no field defaults today and v1 refuses to grow +compiler surface for a value the program can backfill itself. + +**Everything else refuses, precisely.** Refusal messages follow the +`resident: keys` house style: name the class, the field, the stored shape, +the compiled shape, and what to do about it. + +- A same-named field with a different kind: refused — no implicit + conversions. +- A deleted field and an added field of the same kind in one step: refused — + byte-for-byte indistinguishable from a rename, and one interpretation + destroys data the other preserves. The message says to deploy the delete + and the add as two separate steps when both are genuinely meant. + (`@renamed_from` is the v2 answer.) +- A stored class absent from the compiled program: refused — its rows demand + a verdict the runtime must not guess. (The existing durable-to-volatile + refusal from databasev2 2 stays as-is.) +- Index or storage-flag changes: refused in v1. + +## Mechanics + +**The schema record.** A new WAL record kind, `WO_WAL_SCHEMA`, CRC-framed +like every other record, describing every durable class: class name, storage +flags, and per field its name and kind plus the container/reference metadata, +then the secondary-index layout. It is written as the first record of a fresh +log and as the first record compaction writes, so the head of a log always +states the shape of everything after it. + +**The boot diff.** Before replay, the runtime peeks the log head. + +- No schema record — a legacy log. Replay proceeds against the compiled + shape exactly as today; the next compaction writes the record and the log + is self-describing from then on. A legacy log whose shape already changed + still fails as today: there is nothing recorded to diff against. +- Record present and equal to the compiled shape — normal replay. Equality + is by class NAME and field NAME + kind, so pure declaration reordering is + a match, which closes the silent cid-renumbering hole as a side effect. +- Record present and different — classify per class into migrate or refuse, + as above. + +**The migration pass.** Replay decodes each record with the OLD shape, +reconstructed from the schema record — old kinds drive the value decoder, +old names drive the mapping. Rows are built directly in the new shape +through a per-class field map: surviving fields move to their new slot, +deleted fields' values are freed, added fields take the zero value. Old cids +map to new cids by class name. A delta record's field index is an old-shape +index; the fold happens in old shape and maps afterward, so a delta on a +deleted field folds to a no-op. The pass then writes a fresh log the way +compaction does — schema record, one full-row record per live row, fsync, +rename — and boot continues on the new log. Keys-resident tables get their +offset maps rebuilt from the new log by the machinery compaction already +carries. One stderr line per migrated class reports what changed, rows +rewritten, and values dropped. + +**Crash safety is inherited, not added.** The rename swap is atomic: a crash +anywhere in the pass leaves the old log intact and the next boot re-migrates +from it. This is databasev2 3's mutation-proven design, reused. + +## Acceptance criteria + +- Given a log written with shape A and a binary compiled with shape A plus a + new field, when the program boots, then it runs, the field reads as the + kind's zero value, and the log head carries the new schema. +- Given shape A minus a field, when it boots, then the dropped values are + freed (leak-clean under ASan) and no record of the field remains in the + rewritten log. +- Given a simultaneous delete and add of the same kind, when it boots, then + it refuses naming both fields and the two-step alternative. +- Given a retype or a vanished class with stored rows, when it boots, then it + refuses naming the class, field and both shapes. +- Given pure declaration reordering, when it boots, then no migration runs + and every row lands in its right class. +- Given a keys-resident table across a migration, when read after boot, then + folds resolve through the new offsets and a delta on a deleted field is + gone. +- Given a legacy log with an unchanged shape, when it boots, then replay is + byte-for-byte today's behaviour and the next compaction writes the schema + record. +- Given a kill between the new log's write and the rename, when the next + boot runs, then it re-migrates from the intact old log. + +## Out of scope (v2 candidates, each with its reason) + +- **Rename** — needs declared intent (`@renamed_from`); v1 refuses the + ambiguous diff instead of guessing. +- **Retype / conversions** — even lossless ones invite silent surprises; + refusal costs one explicit backfill program. +- **Dropped-class data verdicts** — refusing keeps the data; deleting it is + a decision, not a migration. +- **Data/seed migrations** (`@seed(n)`) — deferred by decision; the site's + trap 1 stays documented in the deploy guide. +- **Field-default syntax** — new grammar for what zero-fill plus app code + already covers. +- **Index migrations** — indexes are rebuilt at boot anyway; the refusal is + about flags, not data, and can be relaxed later with evidence.