From 94da1b603f29c29dadd233747ea3ef1401620bf2 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Thu, 20 Aug 2026 04:00:27 +0200 Subject: [PATCH] =?UTF-8?q?docs(spec):=20iteration=2018=20=E2=80=94=20tran?= =?UTF-8?q?saction=20{=20}=20+=20cache/flags/jobs=20design?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Part A transaction: one wal_commit at block end over the existing staged batch; reads see own writes (RAM stays authoritative); trap unwinding out = abort (undo list: insert->remove, update/delete-> pre-image, captured before RAM apply, txn-only cost); try inside keeps the block alive; WO-E110 lexical nesting, WO_T_DB dynamic; no new opcodes, no .wob bump (internal builtins + catch-frame-shaped abort marker); E108/E109 stay reserved for parked 17 - Part B: cache.wo (ttl_ms/cap, lazy time.now-ms expiry, FIFO over LRU with the tradeoff stated, Text values via json); flags.wo (@table wf_flags, on as Int 0/1 - Bool columns unproven, read-through map, set updates table+map); jobs.wo (@table wf_jobs, enqueue composes with transaction, JobRunner interface, App.jobs(take r, budget), Dispatcher.idle() called post-accept PRE-PARSE - deterministic for the SIGKILL durability proof, unlike after-response) - web-app demo: transactional order+confirm enqueue, GET /jobs count, POST /flags/:name with a flag-gated header on the product list - gate: SIGKILL-after-201/restart/drain proof + flags persistence; corpus carries transaction-commit/abort + WO-E110 + cache-ttl (stamps injected, no sleeps) - board row 18 -> spec written, awaiting review Co-Authored-By: Claude Opus 5 (1M context) --- docs/00-status.md | 2 +- .../2026-08-20-memory-db-features-design.md | 197 ++++++++++++++++++ 2 files changed, 198 insertions(+), 1 deletion(-) create mode 100644 docs/superpowers/specs/2026-08-20-memory-db-features-design.md diff --git a/docs/00-status.md b/docs/00-status.md index eac54fc..3cb403f 100644 --- a/docs/00-status.md +++ b/docs/00-status.md @@ -172,7 +172,7 @@ that sequences its tasks. Read one, approve, then the next starts. | 15 | [deps: `wo.toml [deps]`](stories/language-runtime-database/15-deps-package-manager.md) | ✅ **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 | | 16 | [web framework](stories/language-runtime-database/16-web-framework.md) | ✅ **landed 2026-08-19** — writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (§C) behind 8/9f/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 | | 17 | [library projects + `internal/`](stories/language-runtime-database/17-library-projects-internal.md) | ⏸ **PARKED 2026-08-20** (developer directive; framework v1 first) — forks settled, spec + plan approved and ready on branch `library-internal`: kind = "library" key; Go internal/ rule, dep-boundary-only; lib+bin dual; VM/GC untouched by design | -| 18 | [memory-rich framework features](stories/language-runtime-database/18-memory-db-features.md) | ⬜ **forks settled 2026-08-20, awaiting spec/plan**: TTL cache + @table flags + durable job queue (drain-on-request, idle-drains-nothing disclosed) + `transaction { }` over the WAL's existing staged batch; pub/sub REJECTED until 8/11 | +| 18 | [memory-rich framework features](stories/language-runtime-database/18-memory-db-features.md) | 🔄 **spec written 2026-08-20, awaiting review** ([spec](superpowers/specs/2026-08-20-memory-db-features-design.md)): TTL cache + @table flags + durable job queue (drain-on-request) + `transaction { }` over the WAL's staged batch; pub/sub REJECTED until 8/11 | --- diff --git a/docs/superpowers/specs/2026-08-20-memory-db-features-design.md b/docs/superpowers/specs/2026-08-20-memory-db-features-design.md new file mode 100644 index 0000000..70477d3 --- /dev/null +++ b/docs/superpowers/specs/2026-08-20-memory-db-features-design.md @@ -0,0 +1,197 @@ +# Iteration 18 — memory-rich features over the embedded database: design + +> **Status: spec, awaiting review (2026-08-20).** Decisions were settled in +> [the iteration](../../stories/language-runtime-database/18-memory-db-features.md); +> this spec makes them buildable. The plan follows after review. +> Board: [docs/00-status.md](../../00-status.md). +> +> Per repo convention: concept, reason, and required behavior in words +> only — no implementation code. + +## Goal + +Four pieces, one theme — the single binary's memory and its durable store +are the same process, so features other stacks assemble from Redis, a +broker, and an outbox pattern become a class, a table, and one language +block: a TTL cache (`framework/cache.wo`), `@table`-backed feature flags +with a cached read (`framework/flags.wo`), a durable job queue drained +in-process (`framework/jobs.wo` + a serve-loop seam), and `transaction { }` +exposing the WAL's existing staged batch so a job enqueue and the business +write it belongs to are ONE commit. + +## Part A — `transaction { }` (the one language + engine seam) + +### Observable semantics (normative) + +- `transaction { }` is a statement. Every `insert`, `update` + (field assignment on a table row), and `delete` inside the block becomes + durable together: exactly one WAL write + fdatasync at the closing + brace. Before that point, none of it is durable. +- Reads inside the block see the block's own writes (RAM stays applied + immediately — the engine's RAM-authoritative doctrine is unchanged). +- A trap that unwinds OUT of the block aborts it: the staged WAL batch is + discarded and every RAM effect of the block is undone — rows inserted + are removed (indexes included), updated rows revert to their pre-images, + deleted rows are restored. The trap then continues to the enclosing + handler exactly as it would have without the block; `try` INSIDE the + block that catches a trap keeps the transaction alive (statement-level + failure, e.g. a WO_T_UNIQUE insert, stages nothing for that statement — + same as today). +- Commit failure (WAL write/fdatasync error at the closing brace) is the + existing engine failure trap; the batch stays staged per the WAL + contract and the abort path above runs as the trap unwinds. +- Nesting is rejected at compile time: a `transaction { }` lexically or + dynamically inside another is **WO-E110** (lexical nesting is a parse + check; a transactional function called inside a block traps WO_T_DB + "nested transaction" at run time — the compiler cannot see across + calls, the VM can). WO-E108/E109 stay reserved for parked iteration 17. +- An empty block commits nothing and costs no syscall (the WAL's + empty-batch rule, already in the contract). +- Crash between commit and anything else: recovery replays the WAL — + either the whole block's rows exist or none do. This is the acceptance + criterion's SIGKILL proof. + +### Engine seam (`database/src`) + +`db.c` today calls `wal_append_*` then `wo_wal_commit` per statement — the +staging machinery is already transactional in shape. The change: a +transaction-depth flag on the db handle; while set, statements append but +do NOT commit; the block's end commits once. Abort needs pre-images: while +the flag is set, the engine records an undo entry per statement BEFORE the +RAM apply (insert → the new row id, to remove; update → a copy of the row +before the change; delete → a copy of the removed row, to restore, index +entries included). Abort walks the undo list in reverse, then discards the +staged batch. The undo list exists only while a transaction is open — +zero cost otherwise. + +### VM + compiler surface + +No new opcodes, no `.wob` version bump: the block lowers to +compiler-emitted internal builtins (begin / commit) that user code cannot +name, plus an abort marker on the trap-unwind path — the catch-frame +machinery already unwinds regions; a transaction region behaves like a +catch frame whose only action is "abort the transaction, keep unwinding". +The parser adds the `transaction` keyword and the WO-E110 nesting check; +the ownership and GC passes see an ordinary block. + +## Part B — the framework pieces (pure `.wo`) + +Framework-owned tables use the `wf_` name prefix — a dependency's tables +land in the consuming app's database, so the prefix marks whose they are +(disclosed in the framework README). + +### `framework/cache.wo` — TTL + capacity cache + +A `Cache` class the app holds as a field on any long-lived instance +(App, a middleware, a handler): `ttl_ms: Int`, `cap: Int`, insertion-order +key list, value map, stamp map (`time.now` is wall-clock milliseconds). +`get(key) -> ?Text`: nil when absent or older than ttl_ms (the expired +entry is removed on that read — lazy expiry, there are no timers by +design). `put(key, value)`: stores, stamps, and when size exceeds `cap` +evicts the OLDEST-INSERTED entries until within capacity — FIFO, decided +over LRU: true LRU needs reordering on every read (O(n) in the key list) +for a benefit v1 does not measure; the tradeoff is stated in the file. +Values are Text — the language has no generics; structured values go +through `json.encode`/`decode` (stated in the file). + +### `framework/flags.wo` — feature flags + +`@table(name: "wf_flags")` class `Flag { name @unique, on: Int }` — `on` +is 0/1 because Int columns are the proven storage ground; a Bool column +is not, and flags do not get to be the probe. A `Flags` wrapper class +(held like the cache): `read(name, default) -> Bool` answers from an +in-memory map filled from the table on first read; `set(name, on)` writes +the table (update-or-insert) AND updates the map in the same call — one +process, so "cache invalidation" is an assignment. A restart rebuilds the +map from the table: flags are durable. + +### `framework/jobs.wo` + the drain seam — background jobs + +- `@table(name: "wf_jobs")` class + `Job { kind: Text, payload: Text, attempts: Int, not_before: Int }` + (`not_before` in `time.now` milliseconds; 0 = immediately due). +- `enqueue(kind, payload)` inserts a due job — called by app code, and + called INSIDE `transaction { }` next to the business write it belongs + to; that composition is the point of Part A. +- `JobRunner` interface: `fn run(kind: Text, payload: Text) -> Bool` — + true = done, false = keep. The app implements it as a class (the + Handler doctrine), dispatching on `kind` itself in v1. +- Registration: `App` gains `jobs(take r: Jr, budget: Int)` (`Jr` wraps + the interface value, the `Mw`/`Route` pattern). No registration = the + seam costs nothing. +- The seam: the `Dispatcher` interface gains `fn idle()`; the serve loop + calls it **after accepting a connection, before parsing its first + request**. That placement is deterministic where "after the response" + is not: a job enqueued by connection A provably does NOT run before A + closes, and a SIGKILL after A's response provably leaves the row — + which is exactly what the durability acceptance needs to observe. The + cost — up to `budget` jobs of latency ahead of the next request — is + the disclosed price of drain-on-request; an IDLE server drains nothing + (iteration decision, restated in the file; fibers (11) replace the + scheduler, the table and interface stay). +- Draining: query up to `budget` due jobs (`not_before <= time.now`, + registration-order `take budget`), each inside `try`: true → the row is + deleted; false or trap → `attempts` increments and the row stays + (retry/backoff policy is app-side in v1 — the app can rewrite + `not_before` from its own runner). + +### `web-app` demonstration + +`CreateOrder` wraps its insert and a `confirm` enqueue in one +`transaction { }`; a runner class answers `confirm` by printing an +order-confirmation line to stderr (observable in the gate's server log) +and returning true. `GET /jobs` (behind the existing auth) answers the +pending-job count as JSON — the gate's counting window. Flags demo: +`POST /flags/:name` (auth'd) flips a flag through `Flags.set`, and the +product list answers an extra response header while the flag is on — +small, observable, durable across restart. + +## Gate (`scripts/web-app-accept.sh` + corpus) + +Corpus (single-project fixtures — no manifest needed, so `transaction` +tests live here, unlike 17's): + +- `run/transaction-commit`: two inserts in one block; both rows readable + after. +- `run/transaction-abort`: a block whose second insert traps + (WO_T_UNIQUE); after the trap is caught OUTSIDE the block, the FIRST + insert's row must be gone too, and inserts after the abort still work. +- `compile-fail/transaction-nested`: lexical nesting is WO-E110. + +Gate additions (order matters): + +1. `POST /orders` answers 201 (the transactional enqueue); `kill -9` + the server immediately; restart; the confirmation line appears in the + restarted server's log on the next request (`GET /jobs` → 0 after the + drain) — the job survived the kill because it committed WITH the order. +2. A flags check: `POST /flags/:name` flips it, the next `GET /products` + carries the flag-gated header, restart, still carries it. +3. The standing matrix stays green; the count goes wherever it lands + (numbers are dynamic in the script). + +The cache class is gate-covered indirectly and probe-covered directly: +its fixture (`run/cache-ttl`) injects stamps rather than sleeping — +expiry logic must be testable without wall-clock waits. + +## Out of scope (restated from the iteration) + +Pub/sub and WebSockets (behind 8/11); job priorities and cron shapes; +retry/backoff policy in the framework; exposing the WAL batch API beyond +`transaction { }`; generic cache value types (no generics in the +language); Bool table columns; multi-node anything. + +## Success criteria + +1. **Given** two inserts in `transaction { }` and SIGKILL before the next + request, **when** the server restarts, **then** both rows exist — + and **given** a trap unwinding out of the block, **then** neither + does, and the process keeps serving (`run/transaction-abort` + + gate check 1). +2. **Given** an order POST, **then** its job runs after the next accepted + connection within budget, never before the posting connection closes, + and survives a kill in between (gate check 1). +3. **Given** an expired or evicted cache entry, **then** `get` answers + nil without any timer having existed (`run/cache-ttl`). +4. **Given** a flag flipped and the process restarted, **then** the flag + holds (gate check 2). All standing gates stay green; VM opcode set and + `.wob` format unchanged.