From 0b0d54121f199cd031a039e145dc1e75db503e3f Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Sun, 30 Aug 2026 06:21:18 +0200 Subject: [PATCH] docs(db2-keys): the residency example becomes a product catalogue MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - the motivating workload was an audit log, which is append-only and so argues for nothing. A catalogue is the real case: stable ids, and stock moving on every order while name/price/sku sit still - Product (durable, resident: all) and Cart (durable: false), with place_order decrementing stock through a write-through field assign - run `order` twice and stock goes 10 -> 7 -> 4: a level below the seeded 10 can only mean an earlier order's UPDATE replayed. That is the stronger claim — not just that inserts survive, but that a field change does - caught by running it three times: my first assertion required before == 10, which only holds on a fresh seed and failed on the third run even though the data was correct - gate gains a leg for the update-replay claim; residency 12/0 - the commented resident: keys block now argues the DESIGN too: only stock changes per sale, so appending the whole row would rewrite every field to move one integer on a shop's hottest write path Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit d4104dcc5e3a460459dbf2d24043ce25b113bcdf) --- docs/examples/residency/README.md | 47 ++++++---- docs/examples/residency/main.wo | 147 ++++++++++++++++++------------ scripts/residency-accept.sh | 12 ++- 3 files changed, 129 insertions(+), 77 deletions(-) diff --git a/docs/examples/residency/README.md b/docs/examples/residency/README.md index b3b6a13..d0162d6 100644 --- a/docs/examples/residency/README.md +++ b/docs/examples/residency/README.md @@ -6,8 +6,8 @@ variable for the whole process. Before it, `WO_DATA` was the only switch. Set, and every table is WAL-logged; unset, and none are. Applications are not uniform — a session table is -disposable, an orders table is precious, and a 120 GB audit table does not fit -in RAM at all. One global switch forces "everything is precious" or "nothing +disposable, a product's stock level is not, and a catalogue large enough to +matter does not fit in RAM at all. One global switch forces "everything is precious" or "nothing is", and you pay for whichever is wrong. ## Run it @@ -26,19 +26,25 @@ one data directory: compiler/_build/default/bin/woc --emit docs/examples/residency/main.wo -o /tmp/residency.wob mkdir -p /tmp/residency-data # WO_DATA must exist; wovm will not create it WO_DATA=/tmp/residency-data runtime/wovm /tmp/residency.wob seed -WO_DATA=/tmp/residency-data runtime/wovm /tmp/residency.wob +WO_DATA=/tmp/residency-data runtime/wovm /tmp/residency.wob order ``` ``` -seeded: orders=3 sessions=3 -after restart: orders=3 sessions=0 -ok: durable replayed, volatile did not +seeded: products=2 carts=1 SKU-1 stock=10 +after restart: products=2 carts=0 +order: SKU-1 stock 10 -> 7 +ok: first order placed; run `order` again to see it replay ``` -The second run inserts nothing. Three orders come back from the log; zero -sessions do, because they were never written to it. Both tables were filled by -the same loop — only the annotation differs, so the difference after the -restart is the annotation's doing and nothing else's. +The second run inserts nothing. Both products come back from the log; the cart +does not, because it was never written to it. Both tables were filled by the +same code — only the annotation differs, so the difference after the restart is +the annotation's doing and nothing else's. + +**Run `order` a third time.** Stock goes 7 → 4, and the example says so: a +level below the seeded 10 can only mean an earlier order's *update* survived a +restart. That is the stronger claim — not just that inserts replay, but that a +field change does. ## The three modes @@ -49,13 +55,22 @@ restart is the annotation's doing and nothing else's. | `resident: all` (default) | every row's payload lives in RAM | ✅ works | | `resident: keys` | the id map stays resident, the payload lives in the WAL and is read back by offset | ⛔ **refused at load** | -## Why `resident: keys` is refused +## Why `resident: keys` is refused — and why this example is the argument -It is the mode the track exists for — a table larger than RAM. Storage, the -read paths, deletes and checkpoint survival all work. **Updating such a row -does not:** the row has no slab slot to mutate, so a write would land in a -scratch buffer and be discarded *silently*. Doing it properly is -read-modify-append. +It is the mode the track exists for: a catalogue is the table that outgrows RAM +first, so `Product` is exactly what you would want to declare keys-resident. +Storage, the read paths, scans, `@unique`, deletes and checkpoint survival all +work. **Updating such a row does not**, and `place_order` is precisely why that +matters — the row has no slab slot to mutate, so the write would land in a +materialised scratch buffer and be discarded *silently*. + +The shape of the fix follows from the same example. When an order is placed +only `stock` changes; `sku`, `name` and `price` do not. Appending the whole row +per sale would rewrite every field to move one integer, on the hottest write +path a shop has — which is the argument for appending a **delta** (id, field, +new value) and folding it on read, with the existing checkpoint doing the fold +that keeps delta chains short. That design is being settled now; the loader +refusal stands until it lands. So the loader refuses the annotation rather than honouring it in name only. Uncomment the `AuditEntry` block in `main.wo` and you get: diff --git a/docs/examples/residency/main.wo b/docs/examples/residency/main.wo index 4ed35a2..d254adf 100644 --- a/docs/examples/residency/main.wo +++ b/docs/examples/residency/main.wo @@ -1,51 +1,63 @@ --- residency — databasev2 2's per-table storage, demonstrated across a --- restart. +-- residency — databasev2 2's per-table storage, demonstrated across a restart +-- with the workload the feature exists for: a product catalogue whose stock +-- moves every time an order is placed. -- -- Before this iteration, durability was ONE environment variable for a whole -- process: WO_DATA set and every @table is WAL-logged, or unset and none are. --- Real applications are not uniform. A session table is disposable, an orders --- table is precious. One global switch forces "everything is precious" or --- "nothing is", and the developer pays for the wrong one. +-- Real applications are not uniform. A cart session is disposable; a product's +-- stock level is not; and a catalogue large enough to matter does not fit in +-- RAM at all. One global switch forces "everything is precious" or "nothing +-- is", and the developer pays for whichever is wrong. -- -- Run it twice against the same WO_DATA directory: -- -- mkdir -p /tmp/residency-data -- WO_DATA must exist already -- WO_DATA=/tmp/residency-data wovm residency.wob seed --- WO_DATA=/tmp/residency-data wovm residency.wob +-- WO_DATA=/tmp/residency-data wovm residency.wob order -- --- The second run inserts nothing. Orders come back from the log; Sessions do --- not, because they were never written to it. +-- The second run places an order. Stock comes back decremented after a +-- restart; the cart does not come back at all. --- Precious: WAL-logged, replayed at boot. `durable: true` is the default, and +-- Precious: WAL-logged, replayed at boot. `durable: true` is the default and -- is written out here only because this example is about the annotation. -@table(name: "orders", index: [sku], durable: true, resident: all) -class Order { +@table(name: "products", index: [sku], durable: true, resident: all) +class Product { sku: Text - cents: Int + name: Text + price: Int + stock: Int } -- Scratch: never written to the WAL, so it costs no disk and no fsync, and it -- is EMPTY after a restart. That is the point — not a bug to work around. -@table(name: "sessions", index: [token], durable: false) -class Session { +@table(name: "carts", index: [token], durable: false) +class Cart { token: Text - uid: Int + sku: Text } --- WHAT DOES NOT COMPILE YET, and why it is written here rather than omitted: +-- WHAT DOES NOT COMPILE YET, and why it is written here rather than omitted. -- --- @table(name: "audit", index: [at], durable: true, resident: keys) --- class AuditEntry { --- at: Int --- what: Text +-- A real catalogue is the table that outgrows RAM first, so `Product` above is +-- exactly what you would want to declare keys-resident: +-- +-- @table(name: "products", index: [sku], durable: true, resident: keys) +-- class Product { +-- sku: Text +-- name: Text +-- price: Int +-- stock: Int -- } -- --- `resident: keys` is the mode for a table too large for RAM: the id map stays --- resident, the row payload lives in the WAL and is read back by offset. The --- storage, the read paths, deletes and checkpoint survival all work today. --- UPDATING such a row does not — the row has no slab slot to mutate, and doing --- it properly means read-modify-append. The loader therefore REFUSES the --- annotation outright rather than honouring it in name only: +-- `resident: keys` keeps the id map in RAM and leaves each row's payload in the +-- WAL, read back by offset. Storage, reads, scans, `@unique`, deletes and +-- checkpoint survival all work today. **Updating such a row does not**, and +-- `place_order` below is precisely why that matters: the row has no slab slot +-- to mutate, so the write would land in a materialised scratch buffer and be +-- discarded SILENTLY. +-- +-- The loader therefore refuses the annotation rather than honouring it in name +-- only: -- -- wovm: class 0 declares `resident: keys`, which is INCOMPLETE: rows are -- stored and read keys-only, but UPDATING one is not implemented (it needs @@ -55,55 +67,74 @@ class Session { -- Note WHERE it is refused: the compiler accepts it and emits a .wob — the -- annotation is a load-time property, so `woc` is green and `wovm` exits 2. -- --- Refusing at load rather than at the first update is deliberate: a developer --- who declared a 120 GB table keys-resident, saw it compile, and shipped would --- discover the gap in production. +-- This example is also the argument for HOW updates should land. Only `stock` +-- changes when an order is placed; `sku`, `name` and `price` do not. Appending +-- the whole row on every sale would rewrite every field to change one integer, +-- on the hottest write path a shop has. -fn count_orders() -> Int { +fn count_products() -> Int { let n = 0; - for _o in from o in Order select o { n = n + 1; } + for _p in from p in Product select p { n = n + 1; } return n; } -fn count_sessions() -> Int { +fn count_carts() -> Int { let n = 0; - for _s in from s in Session select s { n = n + 1; } + for _c in from c in Cart select c { n = n + 1; } return n; } -fn report(label: Text) { - print("${label}: orders=${count_orders()} sessions=${count_sessions()}"); +fn stock_of(sku: Text) -> Int { + for p in from p in Product where p.sku == sku select p { return p.stock; } + return 0 - 1; +} + +-- The update this example exists to show: one field of one row moves, and the +-- other three do not. +fn place_order(sku: Text, qty: Int) -> Int { + for p in from p in Product where p.sku == sku select p { + if p.stock < qty { return 0 - 1; } + p.stock = p.stock - qty; -- writes through to the engine + return p.stock; + } + return 0 - 1; } fn main(args: multi Text) -> Int { if len(args) > 0 { if args[0] == "seed" { - -- Both tables get the same number of rows, from the same code path. - -- Only the annotation differs, so anything that differs after the - -- restart is the annotation's doing and nothing else's. - let i = 0; - while i < 3 { - insert Order { sku: "sku-${i}", cents: 100 + i }; - insert Session { token: "tok-${i}", uid: i }; - i = i + 1; + insert Product { sku: "SKU-1", name: "kettle", price: 2999, stock: 10 }; + insert Product { sku: "SKU-2", name: "mug", price: 799, stock: 40 }; + -- a cart is scratch: same insert, different annotation, different fate + insert Cart { token: "cart-a", sku: "SKU-1" }; + print("seeded: products=${count_products()} carts=${count_carts()} SKU-1 stock=${stock_of("SKU-1")}"); + return 0; + } + if args[0] == "order" { + -- second run: no inserts. Whatever is here came from the log. + let before = stock_of("SKU-1"); + let after = place_order("SKU-1", 3); + print("after restart: products=${count_products()} carts=${count_carts()}"); + print("order: SKU-1 stock ${before} -> ${after}"); + if count_carts() != 0 { + print("UNEXPECTED: a durable:false table survived a restart"); + return 1; + } + if after != before - 3 { + print("UNEXPECTED: the stock update did not apply"); + return 1; + } + -- Run `order` more than once and this is the interesting line: a stock + -- level below the seeded 10 can only mean an EARLIER order's update + -- survived a restart. The decrement is durable, not just the insert. + if before < 10 { + print("ok: an earlier order's decrement replayed from the log"); + } else { + print("ok: first order placed; run `order` again to see it replay"); } - report("seeded"); return 0; } } - - -- Second run: no inserts at all. Whatever is here came from the log. - report("after restart"); - - -- The claim this example exists to make, checked rather than narrated. - if count_sessions() != 0 { - print("UNEXPECTED: a durable:false table survived a restart"); - return 1; - } - if count_orders() == 0 { - print("UNEXPECTED: a durable:true table did not replay"); - return 1; - } - print("ok: durable replayed, volatile did not"); + print("usage: residency seed | residency order"); return 0; } diff --git a/scripts/residency-accept.sh b/scripts/residency-accept.sh index 57332ee..9a6cb44 100755 --- a/scripts/residency-accept.sh +++ b/scripts/residency-accept.sh @@ -150,13 +150,19 @@ if "$WOC" --emit "$EX/main.wo" -o "$WORK/residency.wob" >>"$LOG" 2>&1; then { echo "--------------------- run 1: seed ---------------------" WO_DATA="$WORK/exdata" "$WOVM" "$WORK/residency.wob" seed 2>&1 - echo "--------------------- run 2: restart ------------------" + echo "--------------------- run 2: restart + order ----------" } >> "$LOG" - ex2="$(WO_DATA="$WORK/exdata" "$WOVM" "$WORK/residency.wob" 2>&1)" + ex2="$(WO_DATA="$WORK/exdata" "$WOVM" "$WORK/residency.wob" order 2>&1)" printf '%s\n' "$ex2" >> "$LOG" - grep -q 'orders=3 sessions=0' <<<"$ex2" \ + grep -q 'products=2 carts=0' <<<"$ex2" \ && ok "example: durable replayed, volatile did not" \ || bad "example restart" "got: $ex2" + # the stronger claim: a FIELD CHANGE survives, not just an insert + ex3="$(WO_DATA="$WORK/exdata" "$WOVM" "$WORK/residency.wob" order 2>&1)" + printf '%s\n' "$ex3" >> "$LOG" + grep -q "an earlier order's decrement replayed" <<<"$ex3" \ + && ok "example: a stock update replays across a restart" \ + || bad "example update replay" "got: $ex3" else bad "the doc example compiles" "see $LOG" fi