From 7acd085f4699c21d00f9fa93e0ac87079cc0777d Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Tue, 15 Sep 2026 01:04:20 +0200 Subject: [PATCH] =?UTF-8?q?test(db2-chains):=20the=20oracle=20closes=20dat?= =?UTF-8?q?abasev2=2011's=20last=20criterion=20=E2=80=94=20keys=20vs=20all?= =?UTF-8?q?=20across=203=C3=97K=20flattenings=20and=20a=20replay?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - test_oracle_all_vs_keys_same_update_sequence continues the shared sequence 3×WO_DELTA_MAX_HOPS steps, alternating scalar and Text, and asserts the `resident: all` and `resident: keys` rows equal after EVERY step; the fold's hop count proves the chain terminated at least twice and never exceeded K; then the keys log replays into a fresh store and is compared against the oracle once more — the criterion as written, which the story carried as ⚠ "an expected value, not an oracle table" - wal.h: wo_wal_append_row_image's comment claimed the flattened image is written as WO_WAL_INSERT; it is WO_WAL_UPDATE — an INSERT would replay as a duplicate id; compaction alone writes INSERT, into a FRESH log — as 11 landed it and its story recorded - story 11: the criterion flips to ✅ naming the test; the sequencing note and out-of-scope bullet record task 7's 2026-08-30 measurement (16× vs 105× collapse under a cap, 1.53× faster than swapping) — the work stands - test_wal 6295 → 6880 pass, 0 fail; 21 runtime suites 0 fail Co-Authored-By: Claude Fable 5.1 (cherry picked from commit d841390f3087a0c2542ddf23f25f15037a7d4d71) --- database/src/wal.h | 8 ++- .../databasev2/11-bounded-delta-chains.md | 27 +++++---- runtime/test/test_wal.c | 55 ++++++++++++++++++- 3 files changed, 75 insertions(+), 15 deletions(-) diff --git a/database/src/wal.h b/database/src/wal.h index 7e4fb78..aeedfaf 100644 --- a/database/src/wal.h +++ b/database/src/wal.h @@ -513,9 +513,11 @@ int wo_wal_fold_row_at(wo_wal *w, wo_db *db, uint64_t off, uint32_t *class_out, * rather than one looked up by id. wo_wal_append_insert sources its values via * wo_row_ptr, which is NULL for a keys-resident row whose payload has been * dropped; the update path holds a materialised row and needs to log it as a - * chain-terminating record. Written as WO_WAL_INSERT because that is what a - * chain's base must be: it has to replay into a database where nothing - * precedes it. */ + * chain-terminating record. Written as WO_WAL_UPDATE, not WO_WAL_INSERT: the + * live log already carries the row's insert, so an INSERT here would replay as + * a duplicate id (corruption). UPDATE replays as remove-then-recreate and the + * fold terminates on either full-row kind. (Compaction's own flattening writes + * INSERT because it builds a FRESH log — see wal.c.) */ int wo_wal_append_row_image(wo_wal *w, wo_db *db, uint32_t class_id, uint64_t id, const db_row *r); diff --git a/docs/stories/databasev2/11-bounded-delta-chains.md b/docs/stories/databasev2/11-bounded-delta-chains.md index 3718fd0..79ef944 100644 --- a/docs/stories/databasev2/11-bounded-delta-chains.md +++ b/docs/stories/databasev2/11-bounded-delta-chains.md @@ -111,11 +111,14 @@ All verified but one, which is narrowed rather than dropped. Tests live in - ✅ **Given** the same row, **when** the process restarts, **then** replay is correct and its cost does not grow with the total updates ever applied. *`test_delta_chain_flatten_replays`.* -- ⚠️ **Given** a flattening update, **when** replayed, **then** the row matches +- ✅ **Given** a flattening update, **when** replayed, **then** the row matches the same row in a `resident: all` table under the same update sequence. - *Asserted against an expected value, not against a `resident: all` oracle - table. Weaker than written: it catches a wrong value, but it would not catch - the two modes disagreeing in a way that also fooled the expectation.* + *`test_oracle_all_vs_keys_same_update_sequence` (closed 2026-09-09): the + same sequence runs 3×K further updates, alternating scalar and Text, against + a `resident: all` and a `resident: keys` table, asserting equal rows after + every step; the fold's hop count proves the chain was terminated at least + twice and never exceeded K; then the keys log is replayed into a fresh store + and compared against the oracle once more.* - ✅ **Given** a flattening update to an indexed column, **when** queried through that index, **then** the row is found by its new value and not its old, before and after a restart. *`test_keys_resident_indexed_across_flatten`, checked at @@ -143,8 +146,8 @@ compose, and it now pins that. before. - **A time-based compaction trigger.** Records are durable at commit, so an idle log does not grow. -- **Whether `resident: keys` earns its place at all.** That is iteration 2's - task 7, and it should arguably run *before* this work — see below. +- **Whether `resident: keys` earns its place at all.** That was iteration 2's + task 7, measured 2026-08-30 after this landed — see below. ## Info — the forks, settled @@ -165,8 +168,10 @@ compose, and it now pins that. ## Sequencing note -This iteration is **ready but arguably should not be next**. Iteration 2's -task 7 has still never measured whether `resident: keys` beats the kernel's own -paging, and everything built on it — including this — assumes it does. If that -measurement comes back poorly, this work is optimising something that should be -deleted. Recommended order: measure first, then this. +This iteration was written as **ready but arguably not next**: iteration 2's +task 7 had not yet measured whether `resident: keys` beats the kernel's own +paging, and everything built on it — including this — assumes it does. The +recommended order (measure first) was not followed; task 7 measured on +2026-08-30 and the answer is qualified but positive — under a memory cap +`resident: keys` collapses 16× where `resident: all` collapses 105×, 1.53× +faster than swapping, with a 2.55× smaller resident set — so this work stands. diff --git a/runtime/test/test_wal.c b/runtime/test/test_wal.c index edd915b..618b14e 100644 --- a/runtime/test/test_wal.c +++ b/runtime/test/test_wal.c @@ -3678,9 +3678,62 @@ static void test_oracle_all_vs_keys_same_update_sequence(void) { assert_rows_equal(&db_all, id_all, &db_keys, id_keys, &rt, label); } + /* databasev2 11 closure: the same sequence continued 3×K steps past the + six above, alternating scalar and Text, so the keys chain is TERMINATED + by a full-row image more than once while the oracle keeps mutating its + slab. Equality is re-asserted after every step; the fold's hop count + proves the boundary was crossed, not merely approached. */ + uint32_t peak = 0, resets = 0; + for (uint32_t n = 1; n <= WO_DELTA_MAX_HOPS * 3u; n++) { + uint32_t field = n & 1u; + uint64_t val_all, val_keys; + if (field == 0) { + val_all = val_keys = 100u + n; + } else { + char text[16]; + snprintf(text, sizeof text, "t%u", n); + uint32_t tl = (uint32_t)strlen(text); + val_all = (uint64_t)(uintptr_t)wo_str_new(&rt, text, tl); + val_keys = (uint64_t)(uintptr_t)wo_str_new(&rt, text, tl); + } + int ek = 0; + T_EQ(wo_row_update_field(&db_all, 0, id_all, field, val_all, &msg, &ek), 0); + T_EQ(ek, DB_ERR_NONE); + chain_update(&db_keys, &w, 0, id_keys, field, val_keys); + + uint32_t hops = 0; + uint64_t out[2] = {0, 0}; + const char *fm = ""; + uint64_t o1 = wo_row_offset1(&db_keys, 0, id_keys); + T_CHECK(o1 != 0); + T_EQ(wo_wal_fold_row_at(&w, &db_keys, o1 - 1, NULL, NULL, out, &hops, &fm), 0); + for (uint32_t i = 0; i < 2; i++) wo_db_val_free(&db_keys, KEYS_CLASSES[0].kinds[i], out[i]); + if (hops > peak) peak = hops; + if (hops == 0) resets++; + + char label[32]; + snprintf(label, sizeof label, "flatten step %u", n); + assert_rows_equal(&db_all, id_all, &db_keys, id_keys, &rt, label); + } + T_CHECK(peak <= WO_DELTA_MAX_HOPS); /* the bound held throughout */ + T_CHECK(resets >= 2); /* and was actually crossed, twice */ + + /* and across a restart: replay the keys log into a fresh store and compare + it against the oracle, which never left RAM — the criterion as written */ wo_wal_close(&w); - wo_db_destroy(&db_all); wo_db_destroy(&db_keys); + wo_db db_keys2; + T_EQ(wo_db_init(&db_keys2, KEYS_CLASSES, 1, 0, 1), 0); + db_keys2.rt = &rt; rt.wal = NULL; rt.db = &db_keys2; + T_CHECK(wo_wal_replay(path, &db_keys2) >= 0); + wo_wal w2; + T_EQ(wo_wal_open(&w2, path, 1 << 16), 0); + rt.wal = &w2; + assert_rows_equal(&db_all, id_all, &db_keys2, id_keys, &rt, "after replay"); + + wo_wal_close(&w2); + wo_db_destroy(&db_all); + wo_db_destroy(&db_keys2); wo_rt_destroy(&rt); }