docs(db2-migrate): spec + story for schema migrations v1
- brainstorm settled: declarative and automatic at boot; v1 verbs are add and delete only; data/seed migrations deferred to v2 - added fields zero-fill by kind: the grammar has no field-default syntax and v1 refuses to grow compiler surface for it - same-kind delete+add refuses as a disguised rename; retype and vanished classes refuse by name - schema lives in the log itself: WO_WAL_SCHEMA head record, written by fresh-log open and compaction; name-keyed diff also closes the silent cid-renumbering hole - story is iteration 12, board row added, db2-migrate prefix claimed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 072e007b144ff6689b6ff920ea10665900c1a2ef)
This commit is contained in:
parent
552c129ce3
commit
1b6d633ed1
4 changed files with 214 additions and 0 deletions
|
|
@ -37,6 +37,7 @@ features cannot collide.
|
|||
| `db2-delta` | databasev2 2 — keys-resident updates as WAL delta records | ✅ on `master` 2026-08-30 |
|
||||
| `db2-chains` / `db2-chain` | databasev2 11 — bounding a keys-resident row's delta chain | ✅ on `master` 2026-08-30 |
|
||||
| `site` | the writeonce.de tutorial site | ✅ on `master` 2026-08-30 |
|
||||
| `db2-migrate` | databasev2 12 — schema migrations (add/delete, declarative, auto on boot) | claimed 2026-08-31, work on `dev` |
|
||||
| `site-submodule` | `docs/examples/site` extracted to github.com/shoneyJ/writeonce-site and consumed as a submodule | ✅ on `master` 2026-08-30. Both branches now track the site by revision; an edit to it is a commit in that repo plus a pointer bump here |
|
||||
| `lang41` | runtime: unadopted shard must not impersonate shard 0 | on `dev` (`9dca0b4`); independent of the residency stack, not picked |
|
||||
| `porch-store` | porch store tables, Limiter and Idempotent middleware (Phases A, B, C) | on `dev` (`519d411`, `5b1e82a`, `aee7926`). **In progress**: Phase C was uncommitted work from a parallel session, committed as-is, and calls `json.decode`/`json.encode` with no `use json` import |
|
||||
|
|
|
|||
|
|
@ -996,6 +996,7 @@ the language arc as v1 history.
|
|||
| 9 | [Cross-program tables](databasev2/09-cross-program-tables.md) *(was 20)* | ⏸ hold — attach to a running program's database over local IPC |
|
||||
| 10 | [Keypair attach auth](databasev2/10-keypair-attach-auth.md) *(was 21)* | ⏸ hold — program identity as a keypair; needs 9 |
|
||||
| 11 | [Bounded delta chains](databasev2/11-bounded-delta-chains.md) | ✅ **LANDED 2026-08-30.** A `resident: keys` row's delta chain is bounded in the UPDATE path, because the checkpoint is blind to per-row chain length — it thresholds on whole-log bytes, so one hot row can grow an unbounded chain inside a log that never trips compaction. The fold now reports hop count (free — the walk already visited every hop), and past `WO_DELTA_MAX_HOPS` (16) the update writes a full row image instead of a delta, resetting depth to 0. **Two things the tests corrected.** The flattened image is a `WO_WAL_UPDATE`, not an `INSERT`: the row's original INSERT is already in a live log, so a second one for the same id is a duplicate that replay correctly refuses as corruption — INSERT is right only for compaction, which builds a *fresh* log. And the **proportional ceiling was removed as dead code**: with the absolute term at 64 MiB, garbage large enough to reach a 256 MiB ceiling has already tripped it, so the branch was unreachable. Borrowing both constants from postgres was the wrong inference — PG needs two because it thresholds on *tuples* with its pair at opposite ends (base 50, max 1e8); this thresholds on *bytes*, where one constant does both jobs. Found by trying to write a test for the ceiling and finding no input could reach it. Four tests: depth stays bounded across 2K+2 updates, a flattened chain replays, a delta on an **indexed** column composes with flattening (checked at every step across the bound and after restart — found no product defect), and the policy's absolute term with its boundary. `test_wal` **5700 pass / 0 fail**; `wovm-test` and `woc-test` green. **One criterion is weaker than written:** the replay check asserts an expected value, not a `resident: all` oracle table. [spec](../superpowers/specs/2026-08-30-bounded-delta-chains-design.md) |
|
||||
| 12 | [Schema migrations](databasev2/12-schema-migrations.md) | 🔄 **in progress (brainstormed + spec'd 2026-08-31).** A `@table` class is the schema, the log is the database, and today nothing compares them: an added or deleted field turns a healthy `WO_DATA` into "corruption" at boot, and reordering declarations silently decodes rows into the wrong class (records name classes by declaration index). v1, locked: **declarative and automatic at boot** — a `WO_WAL_SCHEMA` record at the log head states the shape; boot diffs it against the compiled classes by NAME; add and delete migrate through a compaction-style rewrite (added fields zero-filled — the grammar has no default syntax; deleted values freed at the rename swap); retype, same-kind delete+add (a disguised rename), and vanished classes REFUSE by name. Data/seed migrations deferred to v2. [spec](../superpowers/specs/2026-08-31-schema-migrations-design.md) |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
77
docs/stories/databasev2/12-schema-migrations.md
Normal file
77
docs/stories/databasev2/12-schema-migrations.md
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
---
|
||||
track: databasev2
|
||||
iteration: "12"
|
||||
status: in-progress
|
||||
readiness: ready
|
||||
---
|
||||
|
||||
# databasev2 12 — schema migrations: add + delete, declarative, auto on boot
|
||||
|
||||
> Part of [Story — databasev2: the database beyond RAM](00-story.md).
|
||||
> Spec: [`2026-08-31-schema-migrations-design.md`](../../superpowers/specs/2026-08-31-schema-migrations-design.md).
|
||||
>
|
||||
> A `@table` class is the schema and the log is the database, but nothing
|
||||
> compares them: today an added or deleted field turns a healthy `WO_DATA`
|
||||
> into "corruption" at boot, and reordering declarations decodes rows into
|
||||
> the wrong class without a sound. Changing the class IS a migration — this
|
||||
> iteration makes the runtime perform the two unambiguous ones and refuse
|
||||
> the rest by name.
|
||||
|
||||
## Decisions (locked in brainstorm, 2026-08-31)
|
||||
|
||||
- **Declarative, automatic, at boot** — the binary carries the schema; no
|
||||
migration files, no offline tool. Chosen over migration-code-in-language
|
||||
and an offline `--migrate` step.
|
||||
- **v1 verbs: add + delete only.** Added fields take the kind's zero value —
|
||||
the grammar has no field-default syntax and v1 does not grow compiler
|
||||
surface. Deleted fields' values are freed and physically dropped at the
|
||||
rename swap.
|
||||
- **Same-kind delete+add in one step refuses** — indistinguishable from a
|
||||
rename, and the wrong guess destroys data. Two-step deploy or v2's
|
||||
`@renamed_from`.
|
||||
- **Data/seed migrations are v2** — the deploy guide's trap 1 (seed only
|
||||
fills an empty table) stays documented, not fixed.
|
||||
|
||||
## Tasks
|
||||
|
||||
| # | Task | State |
|
||||
| --- | --- | --- |
|
||||
| 1 | `WO_WAL_SCHEMA` record: encode/decode, emitted by fresh-log open and by compaction | ⬜ |
|
||||
| 2 | boot diff: peek head, match by class/field NAME, classify migrate / refuse / legacy | ⬜ |
|
||||
| 3 | migration pass: old-shape replay through a field map, compaction-style rewrite, rename swap | ⬜ |
|
||||
| 4 | refusals: retype, same-kind delete+add, vanished class — house-style messages | ⬜ |
|
||||
| 5 | tests: the acceptance list, incl. keys-resident, legacy log, crash injection | ⬜ |
|
||||
| 6 | close out: story, status board, deploy-guide note | ⬜ |
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **Given** a log written with shape A and a binary with one added field,
|
||||
**when** it boots, **then** it runs, the field reads zero-valued, and the
|
||||
log head carries the new schema.
|
||||
- **Given** one deleted field, **when** it boots, **then** dropped values are
|
||||
freed (ASan-clean) and absent from the rewritten log.
|
||||
- **Given** delete+add of the same kind, **when** it boots, **then** refusal
|
||||
naming both fields and the two-step alternative.
|
||||
- **Given** a retype or a vanished class with rows, **when** it boots,
|
||||
**then** refusal naming class, field and both shapes.
|
||||
- **Given** pure declaration reordering, **when** it boots, **then** no
|
||||
migration and every row in its right class *(closes the silent
|
||||
cid-renumbering hole)*.
|
||||
- **Given** a keys-resident table across a migration, **when** read,
|
||||
**then** folds resolve through new offsets; a delta on a deleted field is
|
||||
gone.
|
||||
- **Given** a legacy log (no schema record) with an unchanged shape,
|
||||
**when** it boots, **then** today's behaviour byte-for-byte, and the next
|
||||
compaction writes the record.
|
||||
- **Given** a kill between new-log write and rename, **when** the next boot
|
||||
runs, **then** it re-migrates from the intact old log.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Rename** — v2, via `@renamed_from`; v1 refuses the ambiguous diff.
|
||||
- **Retype/conversions** — refusal costs one explicit backfill program.
|
||||
- **Dropped-class verdicts** — refusing keeps the data; deletion is a
|
||||
decision, not a migration.
|
||||
- **`@seed(n)` data migrations** — deferred by decision.
|
||||
- **Field-default syntax** — zero-fill plus app code covers it.
|
||||
- **Index migrations** — indexes rebuild at boot; refusal is about flags.
|
||||
135
docs/superpowers/specs/2026-08-31-schema-migrations-design.md
Normal file
135
docs/superpowers/specs/2026-08-31-schema-migrations-design.md
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
# Schema migrations v1 — add + delete, declarative, auto on boot
|
||||
|
||||
> Brainstormed and approved 2026-08-31. Story:
|
||||
> [databasev2 12](../../stories/databasev2/12-schema-migrations.md).
|
||||
> Scope settled during brainstorm: declarative model, add/delete only,
|
||||
> zero-value fill, data/seed migrations deferred to v2.
|
||||
|
||||
## The problem
|
||||
|
||||
A `@table` class is the schema and `WO_DATA`'s log is the database, but the
|
||||
two are never compared. A record names its class by `cid` — the class's
|
||||
declaration index — and replay decodes every record against the *compiled*
|
||||
shape. So today, all verified:
|
||||
|
||||
- Adding a field makes every old record under-run its decode and replay
|
||||
refuses the whole log as "corruption".
|
||||
- Deleting a field leaves trailing bytes — the same refusal.
|
||||
- Reordering `@table` declarations renumbers cids and rows decode into the
|
||||
wrong class, silently when shapes happen to match.
|
||||
- Nothing records which schema wrote the log. The durable/resident *mode*
|
||||
mismatch is checked at boot (databasev2 2); the shape is not.
|
||||
|
||||
Changing a `@table` class is a database migration. The language performs it.
|
||||
|
||||
## The model, and why
|
||||
|
||||
**Declarative, automatic, at boot.** The binary already carries the schema —
|
||||
`wo_classdesc` has the class name and per-field names (v2 metadata), so no
|
||||
`.wob` format change is needed. At startup the runtime compares the shape
|
||||
recorded in the log against the compiled shape and either replays (match),
|
||||
migrates (add/delete), or refuses (everything else). No migration files, no
|
||||
offline step: writeonce's doctrine is one binary and no ops choreography, and
|
||||
a migration artifact would be a second source of truth for a schema the class
|
||||
declaration already states.
|
||||
|
||||
**v1 verbs: add and delete only.** They are the two changes whose data
|
||||
consequence is unambiguous: an added field has no stored values and takes the
|
||||
kind's zero value (Int 0, Float 0.0, Bool false, Text empty, optional nil);
|
||||
a deleted field's stored values are freed and physically dropped when the
|
||||
rewritten log is swapped in. Zero-fill is chosen over a default-value syntax
|
||||
because the grammar has no field defaults today and v1 refuses to grow
|
||||
compiler surface for a value the program can backfill itself.
|
||||
|
||||
**Everything else refuses, precisely.** Refusal messages follow the
|
||||
`resident: keys` house style: name the class, the field, the stored shape,
|
||||
the compiled shape, and what to do about it.
|
||||
|
||||
- A same-named field with a different kind: refused — no implicit
|
||||
conversions.
|
||||
- A deleted field and an added field of the same kind in one step: refused —
|
||||
byte-for-byte indistinguishable from a rename, and one interpretation
|
||||
destroys data the other preserves. The message says to deploy the delete
|
||||
and the add as two separate steps when both are genuinely meant.
|
||||
(`@renamed_from` is the v2 answer.)
|
||||
- A stored class absent from the compiled program: refused — its rows demand
|
||||
a verdict the runtime must not guess. (The existing durable-to-volatile
|
||||
refusal from databasev2 2 stays as-is.)
|
||||
- Index or storage-flag changes: refused in v1.
|
||||
|
||||
## Mechanics
|
||||
|
||||
**The schema record.** A new WAL record kind, `WO_WAL_SCHEMA`, CRC-framed
|
||||
like every other record, describing every durable class: class name, storage
|
||||
flags, and per field its name and kind plus the container/reference metadata,
|
||||
then the secondary-index layout. It is written as the first record of a fresh
|
||||
log and as the first record compaction writes, so the head of a log always
|
||||
states the shape of everything after it.
|
||||
|
||||
**The boot diff.** Before replay, the runtime peeks the log head.
|
||||
|
||||
- No schema record — a legacy log. Replay proceeds against the compiled
|
||||
shape exactly as today; the next compaction writes the record and the log
|
||||
is self-describing from then on. A legacy log whose shape already changed
|
||||
still fails as today: there is nothing recorded to diff against.
|
||||
- Record present and equal to the compiled shape — normal replay. Equality
|
||||
is by class NAME and field NAME + kind, so pure declaration reordering is
|
||||
a match, which closes the silent cid-renumbering hole as a side effect.
|
||||
- Record present and different — classify per class into migrate or refuse,
|
||||
as above.
|
||||
|
||||
**The migration pass.** Replay decodes each record with the OLD shape,
|
||||
reconstructed from the schema record — old kinds drive the value decoder,
|
||||
old names drive the mapping. Rows are built directly in the new shape
|
||||
through a per-class field map: surviving fields move to their new slot,
|
||||
deleted fields' values are freed, added fields take the zero value. Old cids
|
||||
map to new cids by class name. A delta record's field index is an old-shape
|
||||
index; the fold happens in old shape and maps afterward, so a delta on a
|
||||
deleted field folds to a no-op. The pass then writes a fresh log the way
|
||||
compaction does — schema record, one full-row record per live row, fsync,
|
||||
rename — and boot continues on the new log. Keys-resident tables get their
|
||||
offset maps rebuilt from the new log by the machinery compaction already
|
||||
carries. One stderr line per migrated class reports what changed, rows
|
||||
rewritten, and values dropped.
|
||||
|
||||
**Crash safety is inherited, not added.** The rename swap is atomic: a crash
|
||||
anywhere in the pass leaves the old log intact and the next boot re-migrates
|
||||
from it. This is databasev2 3's mutation-proven design, reused.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Given a log written with shape A and a binary compiled with shape A plus a
|
||||
new field, when the program boots, then it runs, the field reads as the
|
||||
kind's zero value, and the log head carries the new schema.
|
||||
- Given shape A minus a field, when it boots, then the dropped values are
|
||||
freed (leak-clean under ASan) and no record of the field remains in the
|
||||
rewritten log.
|
||||
- Given a simultaneous delete and add of the same kind, when it boots, then
|
||||
it refuses naming both fields and the two-step alternative.
|
||||
- Given a retype or a vanished class with stored rows, when it boots, then it
|
||||
refuses naming the class, field and both shapes.
|
||||
- Given pure declaration reordering, when it boots, then no migration runs
|
||||
and every row lands in its right class.
|
||||
- Given a keys-resident table across a migration, when read after boot, then
|
||||
folds resolve through the new offsets and a delta on a deleted field is
|
||||
gone.
|
||||
- Given a legacy log with an unchanged shape, when it boots, then replay is
|
||||
byte-for-byte today's behaviour and the next compaction writes the schema
|
||||
record.
|
||||
- Given a kill between the new log's write and the rename, when the next
|
||||
boot runs, then it re-migrates from the intact old log.
|
||||
|
||||
## Out of scope (v2 candidates, each with its reason)
|
||||
|
||||
- **Rename** — needs declared intent (`@renamed_from`); v1 refuses the
|
||||
ambiguous diff instead of guessing.
|
||||
- **Retype / conversions** — even lossless ones invite silent surprises;
|
||||
refusal costs one explicit backfill program.
|
||||
- **Dropped-class data verdicts** — refusing keeps the data; deleting it is
|
||||
a decision, not a migration.
|
||||
- **Data/seed migrations** (`@seed(n)`) — deferred by decision; the site's
|
||||
trap 1 stays documented in the deploy guide.
|
||||
- **Field-default syntax** — new grammar for what zero-fill plus app code
|
||||
already covers.
|
||||
- **Index migrations** — indexes are rebuilt at boot anyway; the refusal is
|
||||
about flags, not data, and can be relaxed later with evidence.
|
||||
Loading…
Reference in a new issue