docs(spec): iteration 18 — transaction { } + cache/flags/jobs design

- Part A transaction: one wal_commit at block end over the existing
  staged batch; reads see own writes (RAM stays authoritative); trap
  unwinding out = abort (undo list: insert->remove, update/delete->
  pre-image, captured before RAM apply, txn-only cost); try inside
  keeps the block alive; WO-E110 lexical nesting, WO_T_DB dynamic;
  no new opcodes, no .wob bump (internal builtins + catch-frame-shaped
  abort marker); E108/E109 stay reserved for parked 17
- Part B: cache.wo (ttl_ms/cap, lazy time.now-ms expiry, FIFO over LRU
  with the tradeoff stated, Text values via json); flags.wo (@table
  wf_flags, on as Int 0/1 - Bool columns unproven, read-through map,
  set updates table+map); jobs.wo (@table wf_jobs, enqueue composes
  with transaction, JobRunner interface, App.jobs(take r, budget),
  Dispatcher.idle() called post-accept PRE-PARSE - deterministic for
  the SIGKILL durability proof, unlike after-response)
- web-app demo: transactional order+confirm enqueue, GET /jobs count,
  POST /flags/:name with a flag-gated header on the product list
- gate: SIGKILL-after-201/restart/drain proof + flags persistence;
  corpus carries transaction-commit/abort + WO-E110 + cache-ttl
  (stamps injected, no sleeps)
- board row 18 -> spec written, awaiting review

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
shoney.arickathil 2026-08-20 04:00:27 +02:00
parent deb2dc805c
commit 24ff960950
2 changed files with 198 additions and 1 deletions

View file

@ -172,7 +172,7 @@ that sequences its tasks. Read one, approve, then the next starts.
| 15 | [deps: `wo.toml [deps]`](stories/language-runtime-database/15-deps-package-manager.md) | ✅ **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 |
| 16 | [web framework](stories/language-runtime-database/16-web-framework.md) | ✅ **landed 2026-08-19** — writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (§C) behind 8/9f/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 |
| 17 | [library projects + `internal/`](stories/language-runtime-database/17-library-projects-internal.md) | ⏸ **PARKED 2026-08-20** (developer directive; framework v1 first) — forks settled, spec + plan approved and ready on branch `library-internal`: kind = "library" key; Go internal/ rule, dep-boundary-only; lib+bin dual; VM/GC untouched by design |
| 18 | [memory-rich framework features](stories/language-runtime-database/18-memory-db-features.md) | ⬜ **forks settled 2026-08-20, awaiting spec/plan**: TTL cache + @table flags + durable job queue (drain-on-request, idle-drains-nothing disclosed) + `transaction { }` over the WAL's existing staged batch; pub/sub REJECTED until 8/11 |
| 18 | [memory-rich framework features](stories/language-runtime-database/18-memory-db-features.md) | 🔄 **spec written 2026-08-20, awaiting review** ([spec](superpowers/specs/2026-08-20-memory-db-features-design.md)): TTL cache + @table flags + durable job queue (drain-on-request) + `transaction { }` over the WAL's staged batch; pub/sub REJECTED until 8/11 |
---

View file

@ -0,0 +1,197 @@
# Iteration 18 — memory-rich features over the embedded database: design
> **Status: spec, awaiting review (2026-08-20).** Decisions were settled in
> [the iteration](../../stories/language-runtime-database/18-memory-db-features.md);
> this spec makes them buildable. The plan follows after review.
> Board: [docs/00-status.md](../../00-status.md).
>
> Per repo convention: concept, reason, and required behavior in words
> only — no implementation code.
## Goal
Four pieces, one theme — the single binary's memory and its durable store
are the same process, so features other stacks assemble from Redis, a
broker, and an outbox pattern become a class, a table, and one language
block: a TTL cache (`framework/cache.wo`), `@table`-backed feature flags
with a cached read (`framework/flags.wo`), a durable job queue drained
in-process (`framework/jobs.wo` + a serve-loop seam), and `transaction { }`
exposing the WAL's existing staged batch so a job enqueue and the business
write it belongs to are ONE commit.
## Part A — `transaction { }` (the one language + engine seam)
### Observable semantics (normative)
- `transaction { <statements> }` is a statement. Every `insert`, `update`
(field assignment on a table row), and `delete` inside the block becomes
durable together: exactly one WAL write + fdatasync at the closing
brace. Before that point, none of it is durable.
- Reads inside the block see the block's own writes (RAM stays applied
immediately — the engine's RAM-authoritative doctrine is unchanged).
- A trap that unwinds OUT of the block aborts it: the staged WAL batch is
discarded and every RAM effect of the block is undone — rows inserted
are removed (indexes included), updated rows revert to their pre-images,
deleted rows are restored. The trap then continues to the enclosing
handler exactly as it would have without the block; `try` INSIDE the
block that catches a trap keeps the transaction alive (statement-level
failure, e.g. a WO_T_UNIQUE insert, stages nothing for that statement —
same as today).
- Commit failure (WAL write/fdatasync error at the closing brace) is the
existing engine failure trap; the batch stays staged per the WAL
contract and the abort path above runs as the trap unwinds.
- Nesting is rejected at compile time: a `transaction { }` lexically or
dynamically inside another is **WO-E110** (lexical nesting is a parse
check; a transactional function called inside a block traps WO_T_DB
"nested transaction" at run time — the compiler cannot see across
calls, the VM can). WO-E108/E109 stay reserved for parked iteration 17.
- An empty block commits nothing and costs no syscall (the WAL's
empty-batch rule, already in the contract).
- Crash between commit and anything else: recovery replays the WAL —
either the whole block's rows exist or none do. This is the acceptance
criterion's SIGKILL proof.
### Engine seam (`database/src`)
`db.c` today calls `wal_append_*` then `wo_wal_commit` per statement — the
staging machinery is already transactional in shape. The change: a
transaction-depth flag on the db handle; while set, statements append but
do NOT commit; the block's end commits once. Abort needs pre-images: while
the flag is set, the engine records an undo entry per statement BEFORE the
RAM apply (insert → the new row id, to remove; update → a copy of the row
before the change; delete → a copy of the removed row, to restore, index
entries included). Abort walks the undo list in reverse, then discards the
staged batch. The undo list exists only while a transaction is open —
zero cost otherwise.
### VM + compiler surface
No new opcodes, no `.wob` version bump: the block lowers to
compiler-emitted internal builtins (begin / commit) that user code cannot
name, plus an abort marker on the trap-unwind path — the catch-frame
machinery already unwinds regions; a transaction region behaves like a
catch frame whose only action is "abort the transaction, keep unwinding".
The parser adds the `transaction` keyword and the WO-E110 nesting check;
the ownership and GC passes see an ordinary block.
## Part B — the framework pieces (pure `.wo`)
Framework-owned tables use the `wf_` name prefix — a dependency's tables
land in the consuming app's database, so the prefix marks whose they are
(disclosed in the framework README).
### `framework/cache.wo` — TTL + capacity cache
A `Cache` class the app holds as a field on any long-lived instance
(App, a middleware, a handler): `ttl_ms: Int`, `cap: Int`, insertion-order
key list, value map, stamp map (`time.now` is wall-clock milliseconds).
`get(key) -> ?Text`: nil when absent or older than ttl_ms (the expired
entry is removed on that read — lazy expiry, there are no timers by
design). `put(key, value)`: stores, stamps, and when size exceeds `cap`
evicts the OLDEST-INSERTED entries until within capacity — FIFO, decided
over LRU: true LRU needs reordering on every read (O(n) in the key list)
for a benefit v1 does not measure; the tradeoff is stated in the file.
Values are Text — the language has no generics; structured values go
through `json.encode`/`decode` (stated in the file).
### `framework/flags.wo` — feature flags
`@table(name: "wf_flags")` class `Flag { name @unique, on: Int }` — `on`
is 0/1 because Int columns are the proven storage ground; a Bool column
is not, and flags do not get to be the probe. A `Flags` wrapper class
(held like the cache): `read(name, default) -> Bool` answers from an
in-memory map filled from the table on first read; `set(name, on)` writes
the table (update-or-insert) AND updates the map in the same call — one
process, so "cache invalidation" is an assignment. A restart rebuilds the
map from the table: flags are durable.
### `framework/jobs.wo` + the drain seam — background jobs
- `@table(name: "wf_jobs")` class
`Job { kind: Text, payload: Text, attempts: Int, not_before: Int }`
(`not_before` in `time.now` milliseconds; 0 = immediately due).
- `enqueue(kind, payload)` inserts a due job — called by app code, and
called INSIDE `transaction { }` next to the business write it belongs
to; that composition is the point of Part A.
- `JobRunner` interface: `fn run(kind: Text, payload: Text) -> Bool` —
true = done, false = keep. The app implements it as a class (the
Handler doctrine), dispatching on `kind` itself in v1.
- Registration: `App` gains `jobs(take r: Jr, budget: Int)` (`Jr` wraps
the interface value, the `Mw`/`Route` pattern). No registration = the
seam costs nothing.
- The seam: the `Dispatcher` interface gains `fn idle()`; the serve loop
calls it **after accepting a connection, before parsing its first
request**. That placement is deterministic where "after the response"
is not: a job enqueued by connection A provably does NOT run before A
closes, and a SIGKILL after A's response provably leaves the row —
which is exactly what the durability acceptance needs to observe. The
cost — up to `budget` jobs of latency ahead of the next request — is
the disclosed price of drain-on-request; an IDLE server drains nothing
(iteration decision, restated in the file; fibers (11) replace the
scheduler, the table and interface stay).
- Draining: query up to `budget` due jobs (`not_before <= time.now`,
registration-order `take budget`), each inside `try`: true → the row is
deleted; false or trap → `attempts` increments and the row stays
(retry/backoff policy is app-side in v1 — the app can rewrite
`not_before` from its own runner).
### `web-app` demonstration
`CreateOrder` wraps its insert and a `confirm` enqueue in one
`transaction { }`; a runner class answers `confirm` by printing an
order-confirmation line to stderr (observable in the gate's server log)
and returning true. `GET /jobs` (behind the existing auth) answers the
pending-job count as JSON — the gate's counting window. Flags demo:
`POST /flags/:name` (auth'd) flips a flag through `Flags.set`, and the
product list answers an extra response header while the flag is on —
small, observable, durable across restart.
## Gate (`scripts/web-app-accept.sh` + corpus)
Corpus (single-project fixtures — no manifest needed, so `transaction`
tests live here, unlike 17's):
- `run/transaction-commit`: two inserts in one block; both rows readable
after.
- `run/transaction-abort`: a block whose second insert traps
(WO_T_UNIQUE); after the trap is caught OUTSIDE the block, the FIRST
insert's row must be gone too, and inserts after the abort still work.
- `compile-fail/transaction-nested`: lexical nesting is WO-E110.
Gate additions (order matters):
1. `POST /orders` answers 201 (the transactional enqueue); `kill -9`
the server immediately; restart; the confirmation line appears in the
restarted server's log on the next request (`GET /jobs` → 0 after the
drain) — the job survived the kill because it committed WITH the order.
2. A flags check: `POST /flags/:name` flips it, the next `GET /products`
carries the flag-gated header, restart, still carries it.
3. The standing matrix stays green; the count goes wherever it lands
(numbers are dynamic in the script).
The cache class is gate-covered indirectly and probe-covered directly:
its fixture (`run/cache-ttl`) injects stamps rather than sleeping —
expiry logic must be testable without wall-clock waits.
## Out of scope (restated from the iteration)
Pub/sub and WebSockets (behind 8/11); job priorities and cron shapes;
retry/backoff policy in the framework; exposing the WAL batch API beyond
`transaction { }`; generic cache value types (no generics in the
language); Bool table columns; multi-node anything.
## Success criteria
1. **Given** two inserts in `transaction { }` and SIGKILL before the next
request, **when** the server restarts, **then** both rows exist —
and **given** a trap unwinding out of the block, **then** neither
does, and the process keeps serving (`run/transaction-abort` +
gate check 1).
2. **Given** an order POST, **then** its job runs after the next accepted
connection within budget, never before the posting connection closes,
and survives a kill in between (gate check 1).
3. **Given** an expired or evicted cache entry, **then** `get` answers
nil without any timer having existed (`run/cache-ttl`).
4. **Given** a flag flipped and the process restarted, **then** the flag
holds (gate check 2). All standing gates stay green; VM opcode set and
`.wob` format unchanged.