docs(spec): iteration 18 — transaction { } + cache/flags/jobs design
- Part A transaction: one wal_commit at block end over the existing staged batch; reads see own writes (RAM stays authoritative); trap unwinding out = abort (undo list: insert->remove, update/delete-> pre-image, captured before RAM apply, txn-only cost); try inside keeps the block alive; WO-E110 lexical nesting, WO_T_DB dynamic; no new opcodes, no .wob bump (internal builtins + catch-frame-shaped abort marker); E108/E109 stay reserved for parked 17 - Part B: cache.wo (ttl_ms/cap, lazy time.now-ms expiry, FIFO over LRU with the tradeoff stated, Text values via json); flags.wo (@table wf_flags, on as Int 0/1 - Bool columns unproven, read-through map, set updates table+map); jobs.wo (@table wf_jobs, enqueue composes with transaction, JobRunner interface, App.jobs(take r, budget), Dispatcher.idle() called post-accept PRE-PARSE - deterministic for the SIGKILL durability proof, unlike after-response) - web-app demo: transactional order+confirm enqueue, GET /jobs count, POST /flags/:name with a flag-gated header on the product list - gate: SIGKILL-after-201/restart/drain proof + flags persistence; corpus carries transaction-commit/abort + WO-E110 + cache-ttl (stamps injected, no sleeps) - board row 18 -> spec written, awaiting review Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
72ee349700
commit
94da1b603f
2 changed files with 198 additions and 1 deletions
|
|
@ -172,7 +172,7 @@ that sequences its tasks. Read one, approve, then the next starts.
|
|||
| 15 | [deps: `wo.toml [deps]`](stories/language-runtime-database/15-deps-package-manager.md) | ✅ **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 |
|
||||
| 16 | [web framework](stories/language-runtime-database/16-web-framework.md) | ✅ **landed 2026-08-19** — writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (§C) behind 8/9f/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 |
|
||||
| 17 | [library projects + `internal/`](stories/language-runtime-database/17-library-projects-internal.md) | ⏸ **PARKED 2026-08-20** (developer directive; framework v1 first) — forks settled, spec + plan approved and ready on branch `library-internal`: kind = "library" key; Go internal/ rule, dep-boundary-only; lib+bin dual; VM/GC untouched by design |
|
||||
| 18 | [memory-rich framework features](stories/language-runtime-database/18-memory-db-features.md) | ⬜ **forks settled 2026-08-20, awaiting spec/plan**: TTL cache + @table flags + durable job queue (drain-on-request, idle-drains-nothing disclosed) + `transaction { }` over the WAL's existing staged batch; pub/sub REJECTED until 8/11 |
|
||||
| 18 | [memory-rich framework features](stories/language-runtime-database/18-memory-db-features.md) | 🔄 **spec written 2026-08-20, awaiting review** ([spec](superpowers/specs/2026-08-20-memory-db-features-design.md)): TTL cache + @table flags + durable job queue (drain-on-request) + `transaction { }` over the WAL's staged batch; pub/sub REJECTED until 8/11 |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
197
docs/superpowers/specs/2026-08-20-memory-db-features-design.md
Normal file
197
docs/superpowers/specs/2026-08-20-memory-db-features-design.md
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
# Iteration 18 — memory-rich features over the embedded database: design
|
||||
|
||||
> **Status: spec, awaiting review (2026-08-20).** Decisions were settled in
|
||||
> [the iteration](../../stories/language-runtime-database/18-memory-db-features.md);
|
||||
> this spec makes them buildable. The plan follows after review.
|
||||
> Board: [docs/00-status.md](../../00-status.md).
|
||||
>
|
||||
> Per repo convention: concept, reason, and required behavior in words
|
||||
> only — no implementation code.
|
||||
|
||||
## Goal
|
||||
|
||||
Four pieces, one theme — the single binary's memory and its durable store
|
||||
are the same process, so features other stacks assemble from Redis, a
|
||||
broker, and an outbox pattern become a class, a table, and one language
|
||||
block: a TTL cache (`framework/cache.wo`), `@table`-backed feature flags
|
||||
with a cached read (`framework/flags.wo`), a durable job queue drained
|
||||
in-process (`framework/jobs.wo` + a serve-loop seam), and `transaction { }`
|
||||
exposing the WAL's existing staged batch so a job enqueue and the business
|
||||
write it belongs to are ONE commit.
|
||||
|
||||
## Part A — `transaction { }` (the one language + engine seam)
|
||||
|
||||
### Observable semantics (normative)
|
||||
|
||||
- `transaction { <statements> }` is a statement. Every `insert`, `update`
|
||||
(field assignment on a table row), and `delete` inside the block becomes
|
||||
durable together: exactly one WAL write + fdatasync at the closing
|
||||
brace. Before that point, none of it is durable.
|
||||
- Reads inside the block see the block's own writes (RAM stays applied
|
||||
immediately — the engine's RAM-authoritative doctrine is unchanged).
|
||||
- A trap that unwinds OUT of the block aborts it: the staged WAL batch is
|
||||
discarded and every RAM effect of the block is undone — rows inserted
|
||||
are removed (indexes included), updated rows revert to their pre-images,
|
||||
deleted rows are restored. The trap then continues to the enclosing
|
||||
handler exactly as it would have without the block; `try` INSIDE the
|
||||
block that catches a trap keeps the transaction alive (statement-level
|
||||
failure, e.g. a WO_T_UNIQUE insert, stages nothing for that statement —
|
||||
same as today).
|
||||
- Commit failure (WAL write/fdatasync error at the closing brace) is the
|
||||
existing engine failure trap; the batch stays staged per the WAL
|
||||
contract and the abort path above runs as the trap unwinds.
|
||||
- Nesting is rejected at compile time: a `transaction { }` lexically or
|
||||
dynamically inside another is **WO-E110** (lexical nesting is a parse
|
||||
check; a transactional function called inside a block traps WO_T_DB
|
||||
"nested transaction" at run time — the compiler cannot see across
|
||||
calls, the VM can). WO-E108/E109 stay reserved for parked iteration 17.
|
||||
- An empty block commits nothing and costs no syscall (the WAL's
|
||||
empty-batch rule, already in the contract).
|
||||
- Crash between commit and anything else: recovery replays the WAL —
|
||||
either the whole block's rows exist or none do. This is the acceptance
|
||||
criterion's SIGKILL proof.
|
||||
|
||||
### Engine seam (`database/src`)
|
||||
|
||||
`db.c` today calls `wal_append_*` then `wo_wal_commit` per statement — the
|
||||
staging machinery is already transactional in shape. The change: a
|
||||
transaction-depth flag on the db handle; while set, statements append but
|
||||
do NOT commit; the block's end commits once. Abort needs pre-images: while
|
||||
the flag is set, the engine records an undo entry per statement BEFORE the
|
||||
RAM apply (insert → the new row id, to remove; update → a copy of the row
|
||||
before the change; delete → a copy of the removed row, to restore, index
|
||||
entries included). Abort walks the undo list in reverse, then discards the
|
||||
staged batch. The undo list exists only while a transaction is open —
|
||||
zero cost otherwise.
|
||||
|
||||
### VM + compiler surface
|
||||
|
||||
No new opcodes, no `.wob` version bump: the block lowers to
|
||||
compiler-emitted internal builtins (begin / commit) that user code cannot
|
||||
name, plus an abort marker on the trap-unwind path — the catch-frame
|
||||
machinery already unwinds regions; a transaction region behaves like a
|
||||
catch frame whose only action is "abort the transaction, keep unwinding".
|
||||
The parser adds the `transaction` keyword and the WO-E110 nesting check;
|
||||
the ownership and GC passes see an ordinary block.
|
||||
|
||||
## Part B — the framework pieces (pure `.wo`)
|
||||
|
||||
Framework-owned tables use the `wf_` name prefix — a dependency's tables
|
||||
land in the consuming app's database, so the prefix marks whose they are
|
||||
(disclosed in the framework README).
|
||||
|
||||
### `framework/cache.wo` — TTL + capacity cache
|
||||
|
||||
A `Cache` class the app holds as a field on any long-lived instance
|
||||
(App, a middleware, a handler): `ttl_ms: Int`, `cap: Int`, insertion-order
|
||||
key list, value map, stamp map (`time.now` is wall-clock milliseconds).
|
||||
`get(key) -> ?Text`: nil when absent or older than ttl_ms (the expired
|
||||
entry is removed on that read — lazy expiry, there are no timers by
|
||||
design). `put(key, value)`: stores, stamps, and when size exceeds `cap`
|
||||
evicts the OLDEST-INSERTED entries until within capacity — FIFO, decided
|
||||
over LRU: true LRU needs reordering on every read (O(n) in the key list)
|
||||
for a benefit v1 does not measure; the tradeoff is stated in the file.
|
||||
Values are Text — the language has no generics; structured values go
|
||||
through `json.encode`/`decode` (stated in the file).
|
||||
|
||||
### `framework/flags.wo` — feature flags
|
||||
|
||||
`@table(name: "wf_flags")` class `Flag { name @unique, on: Int }` — `on`
|
||||
is 0/1 because Int columns are the proven storage ground; a Bool column
|
||||
is not, and flags do not get to be the probe. A `Flags` wrapper class
|
||||
(held like the cache): `read(name, default) -> Bool` answers from an
|
||||
in-memory map filled from the table on first read; `set(name, on)` writes
|
||||
the table (update-or-insert) AND updates the map in the same call — one
|
||||
process, so "cache invalidation" is an assignment. A restart rebuilds the
|
||||
map from the table: flags are durable.
|
||||
|
||||
### `framework/jobs.wo` + the drain seam — background jobs
|
||||
|
||||
- `@table(name: "wf_jobs")` class
|
||||
`Job { kind: Text, payload: Text, attempts: Int, not_before: Int }`
|
||||
(`not_before` in `time.now` milliseconds; 0 = immediately due).
|
||||
- `enqueue(kind, payload)` inserts a due job — called by app code, and
|
||||
called INSIDE `transaction { }` next to the business write it belongs
|
||||
to; that composition is the point of Part A.
|
||||
- `JobRunner` interface: `fn run(kind: Text, payload: Text) -> Bool` —
|
||||
true = done, false = keep. The app implements it as a class (the
|
||||
Handler doctrine), dispatching on `kind` itself in v1.
|
||||
- Registration: `App` gains `jobs(take r: Jr, budget: Int)` (`Jr` wraps
|
||||
the interface value, the `Mw`/`Route` pattern). No registration = the
|
||||
seam costs nothing.
|
||||
- The seam: the `Dispatcher` interface gains `fn idle()`; the serve loop
|
||||
calls it **after accepting a connection, before parsing its first
|
||||
request**. That placement is deterministic where "after the response"
|
||||
is not: a job enqueued by connection A provably does NOT run before A
|
||||
closes, and a SIGKILL after A's response provably leaves the row —
|
||||
which is exactly what the durability acceptance needs to observe. The
|
||||
cost — up to `budget` jobs of latency ahead of the next request — is
|
||||
the disclosed price of drain-on-request; an IDLE server drains nothing
|
||||
(iteration decision, restated in the file; fibers (11) replace the
|
||||
scheduler, the table and interface stay).
|
||||
- Draining: query up to `budget` due jobs (`not_before <= time.now`,
|
||||
registration-order `take budget`), each inside `try`: true → the row is
|
||||
deleted; false or trap → `attempts` increments and the row stays
|
||||
(retry/backoff policy is app-side in v1 — the app can rewrite
|
||||
`not_before` from its own runner).
|
||||
|
||||
### `web-app` demonstration
|
||||
|
||||
`CreateOrder` wraps its insert and a `confirm` enqueue in one
|
||||
`transaction { }`; a runner class answers `confirm` by printing an
|
||||
order-confirmation line to stderr (observable in the gate's server log)
|
||||
and returning true. `GET /jobs` (behind the existing auth) answers the
|
||||
pending-job count as JSON — the gate's counting window. Flags demo:
|
||||
`POST /flags/:name` (auth'd) flips a flag through `Flags.set`, and the
|
||||
product list answers an extra response header while the flag is on —
|
||||
small, observable, durable across restart.
|
||||
|
||||
## Gate (`scripts/web-app-accept.sh` + corpus)
|
||||
|
||||
Corpus (single-project fixtures — no manifest needed, so `transaction`
|
||||
tests live here, unlike 17's):
|
||||
|
||||
- `run/transaction-commit`: two inserts in one block; both rows readable
|
||||
after.
|
||||
- `run/transaction-abort`: a block whose second insert traps
|
||||
(WO_T_UNIQUE); after the trap is caught OUTSIDE the block, the FIRST
|
||||
insert's row must be gone too, and inserts after the abort still work.
|
||||
- `compile-fail/transaction-nested`: lexical nesting is WO-E110.
|
||||
|
||||
Gate additions (order matters):
|
||||
|
||||
1. `POST /orders` answers 201 (the transactional enqueue); `kill -9`
|
||||
the server immediately; restart; the confirmation line appears in the
|
||||
restarted server's log on the next request (`GET /jobs` → 0 after the
|
||||
drain) — the job survived the kill because it committed WITH the order.
|
||||
2. A flags check: `POST /flags/:name` flips it, the next `GET /products`
|
||||
carries the flag-gated header, restart, still carries it.
|
||||
3. The standing matrix stays green; the count goes wherever it lands
|
||||
(numbers are dynamic in the script).
|
||||
|
||||
The cache class is gate-covered indirectly and probe-covered directly:
|
||||
its fixture (`run/cache-ttl`) injects stamps rather than sleeping —
|
||||
expiry logic must be testable without wall-clock waits.
|
||||
|
||||
## Out of scope (restated from the iteration)
|
||||
|
||||
Pub/sub and WebSockets (behind 8/11); job priorities and cron shapes;
|
||||
retry/backoff policy in the framework; exposing the WAL batch API beyond
|
||||
`transaction { }`; generic cache value types (no generics in the
|
||||
language); Bool table columns; multi-node anything.
|
||||
|
||||
## Success criteria
|
||||
|
||||
1. **Given** two inserts in `transaction { }` and SIGKILL before the next
|
||||
request, **when** the server restarts, **then** both rows exist —
|
||||
and **given** a trap unwinding out of the block, **then** neither
|
||||
does, and the process keeps serving (`run/transaction-abort` +
|
||||
gate check 1).
|
||||
2. **Given** an order POST, **then** its job runs after the next accepted
|
||||
connection within budget, never before the posting connection closes,
|
||||
and survives a kill in between (gate check 1).
|
||||
3. **Given** an expired or evicted cache entry, **then** `get` answers
|
||||
nil without any timer having existed (`run/cache-ttl`).
|
||||
4. **Given** a flag flipped and the process restarted, **then** the flag
|
||||
holds (gate check 2). All standing gates stay green; VM opcode set and
|
||||
`.wob` format unchanged.
|
||||
Loading…
Reference in a new issue