writeonce/docs/superpowers/plans/2026-08-20-framework-v2-memory-features.md

Iteration 18 — framework v2 (transaction{} + cache/flags/jobs): implementation plan

Status: ready to execute (2026-08-20). Board: docs/00-status.md.

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Style rule (user convention): concept, reason, and required behavior in words plus verification commands only — no implementation or test code blocks; the executor writes the code.

Goal: transaction { } makes multi-statement writes one WAL commit (trap = abort), and three framework pieces ride the single process: a TTL cache, @table feature flags with a cached read, and a durable @table job queue drained in-process — the web-app proves the headline (order + job enqueue, ONE commit, SIGKILL-survivable).

Architecture: the engine already stages WAL batches (wal_append_* → wo_wal_commit, per-statement today in database/src/db.c's three write cases); a transaction defers the commit and keeps an undo log for RAM. The VM learns two internal builtins and a transaction-flagged catch frame; the parser one keyword. Cache/flags/jobs are pure .wo in the framework plus one serve-loop seam. No new opcodes, no .wob version bump, GC untouched.

Tech Stack: C11 libc-only (engine + VM), OCaml stdlib-only (woc), pure .wo (framework), bash gates.

Spec: ../specs/2026-08-20-memory-db-features-design.md (approved 2026-08-20, normative). Story: 18-memory-db-features.md.

Global Constraints

• Branch framework-v2 off master; commits local only, never push.
• Exit/trap doctrine unchanged: WO-E110 nested transaction (compile), WO_T_DB "nested transaction" (dynamic, through a call); commit failure is the existing WO_T_IO shape with RAM never ahead of disk.
• Gates that must stay green after every task: just woc-test, just oop-e2e, just deps-accept, just web-app, just log-watcher, just employee — plus the oop-accept ASan clause for anything the VM touches.
• Framework tables carry the wf_ prefix; policy stays app-side.
• time.now is wall-clock MILLISECONDS everywhere in this iteration.

Spec deviations, disclosed up front

1. return/break/continue crossing a transaction { } boundary is rejected at compile time (new WO-E112). The spec is silent on early exit; commit-on-return vs abort-on-return is exactly the ambiguity a v1 must not guess at. A later iteration may define it; today the block has one entrance and one exit (a TRAP still aborts — that path is defined).
2. The cache exposes a time-injected seam (get_at/put_at taking a now-milliseconds argument, with get/put sugar reading time.now) so the corpus fixture injects stamps instead of sleeping — the spec's "stamps injected, not waited" made concrete.
3. Flags are gate-proven, not corpus-proven: a @table needs WO_DATA and a persistent directory, which the corpus harness does not provide; just web-app carries the flags checks (the spec's gate section already put them there).
4. The WAL gains an explicit staged-batch discard (wo_wal_abort): today a failed commit "stays staged" by contract; abort needs to drop the batch deliberately. Same file, same batch machinery, new entry point.

---

Task 1 — engine transactions: defer commit, undo log, abort

Files:

• Modify: database/src/db.h (txn state on wo_db, three new entry points), database/src/db.c (the three write cases ~lines 17–80), database/src/wal.h + wal.c (wo_wal_abort), database/src/table.h + table.c only if the replay-only fixed-id create/index pair needs a non-static wrapper.
• Test: runtime/test/test_txn.c (new, mirroring the existing test_*.c harness shape), wired into the runtime test recipe.

Interfaces:

• Produces: wo_db_txn_begin(db) (0 ok; nonzero = already open — the dynamic-nesting signal), wo_db_txn_commit(db, wal) (the ONE wo_wal_commit; on failure the undo walk runs and the error returns), wo_db_txn_abort(db, wal) (reverse undo walk + wo_wal_abort). Task 3's builtins call exactly these three.

• wo_db grows a transaction flag and an undo log (a growable array of entries: op kind, class id, row id, and for update/delete a copy of the row's bytes — wo_row_ptr + the class's row_size are the copy; the log exists only while a transaction is open, zero cost otherwise).

• The three write cases in wo_builtin_db: when the flag is set, record the undo entry BEFORE the RAM apply (update/delete pre-images; insert records just the new id AFTER apply), still wal_append_*, and SKIP the per-statement wo_wal_commit. Flag clear = byte-identical behavior to today (every existing gate is the proof).

• Statement-level failures inside a transaction change nothing: a unique violation traps before apply and stages nothing (already true — wo_row_insert refuses first); a failed append keeps the pre-existing error shape.

• Abort walks the undo log in REVERSE: inserted row → removed; updated row → bytes restored and index entries fixed the replay way (remove + re-add through the same engine-internal pair wal.c replay uses); deleted row → re-created with its FIXED id and re-indexed (the replay-only create), then wo_wal_abort discards the staged batch.

• Reads inside a transaction need no change: RAM stays applied, so scans/point-reads see the block's own writes for free.

• test_txn.c: begin→insert+insert→commit = both rows + ONE wal flush; begin→insert→abort = zero rows, next insert works; update and delete pre-images restored on abort (indexed column included); begin-while-open refused; abort with an empty log is a no-op.

• Run the runtime test suite + just employee (engine untouched when no txn opens). Commit.

Task 2 — the language surface: keyword, block, WO-E110/E112

Files:

• Modify: compiler/src/token.ml + lexer.ml (KwTransaction), compiler/src/ast.ml (a Transaction statement holding a body), compiler/src/parser.ml (block statement + LEXICAL nesting = WO-E110), compiler/src/types.ml (walk the body; E-code constants), compiler/src/owner.ml (treat as a plain nested scope), compiler/src/dump.ml (labels).

Interfaces:

• Consumes: nothing new.

• Produces: the AST node Task 3 lowers; WO-E110 (parsing prefix, nested block), WO-E112 (parsing prefix, return/break/continue whose jump would cross the block boundary — a loop wholly INSIDE the block keeps its own break/continue).

• Keyword + statement parse; the body is an ordinary statement list. A transaction token while one is already open (parser-tracked depth) is WO-E110 at the inner keyword.

• WO-E112: while parsing the block, a return at any depth, or a break/continue not enclosed by a loop that itself started inside the block, names the rule ("a transaction has one exit; lift the return out or end the block first").

• Types/owner: the body checks exactly like a bare block — no new typing rule (the ownership pass sees a scope; values born inside drop inside, exactly as today).

• Compile-fail fixtures: transaction-nested (WO-E110), transaction-early-return (WO-E112). Verify both + just woc-test (dump labels) + just oop-e2e. Commit.

Task 3 — VM lowering: internal builtins + the abort-on-unwind frame

Files:

• Modify: runtime/src/wob.h (two builtin ids in the internal range), runtime/src/builtin.c (dispatch to Task 1's three entry points), runtime/src/vm.c (transaction-flagged catch frame; the vm_trap walk; unwind/rt-destroy cleanup), compiler/src/emit.ml (lower the Transaction statement), runtime/test/test_unwind.c (frame cleanup on a trap that leaves the whole method).

Interfaces:

• Consumes: Task 1's wo_db_txn_begin/commit/abort, Task 2's AST node.

• Produces: the observable spec semantics — one fdatasync at the closing brace; trap unwinding OUT aborts and keeps unwinding; try INSIDE the block keeps it alive.

• Two builtin ids the emitter emits directly from the Transaction case (no name in any user-callable table — nothing to collide with): begin pushes a TRANSACTION-FLAGGED catch frame (the existing TRY frame machinery with one flag bit) and calls txn_begin (nonzero = the WO_T_DB "nested transaction" trap); end calls txn_commit FIRST and pops the frame only on success — a commit failure traps with the frame still in place, so the abort path below runs and RAM is rolled back (the "RAM never ahead of disk" rule, transaction-sized).

• vm_trap's handler search: a transaction-flagged frame is not a handler — abort the transaction, pop it, CONTINUE searching. An inner try frame sits ABOVE it and catches first (the spec's inner-try-keeps-it-alive rule falls out of frame order, no special case).

• Program exit / rt teardown with a transaction somehow open (a trap that reaches main uncaught) must abort, not leak the undo log.

• Emitter: begin, body statements, end — plus the WO-E112 guarantee from Task 2 meaning no jump ever leaves the region except a trap.

• Corpus: run/transaction-commit (two inserts, both rows readable after — needs the trap corpus's WO_DATA-less shape? No: @table without WO_DATA runs RAM-only with no WAL, which still exercises begin/commit frames; the DURABILITY half lives in Task 6's gate where WO_DATA exists), run/transaction-abort (second insert unique-traps, caught OUTSIDE the block: first row gone too, inserts after the abort work, process exits clean under ASan).

• just oop-e2e (ASan stage covers the new frames) + full battery. Commit.

Task 4 — framework cache: TTL + capacity, pure .wo

Files:

• Create: docs/examples/writeonce-framework/store/cache.wo.
• Test: tests/corpus/run/cache-ttl/ (fixture copies the class inline — corpus fixtures cannot use the framework; the framework file is the same code verified by the framework's standalone compile + Task 7's consumer build).

Interfaces:

• Produces: pub class Cache { ttl_ms: Int, cap: Int, keys: multi Text, vals: map<Text, Text>, stamps: map<Text, Int> } with get_at(now, key) -> ?Text, put_at(now, key, value), and get/put sugar over time.now — the shape apps hold as a field on any long-lived instance.

• get_at: absent → nil; older than ttl_ms → remove the entry (lazy expiry — there are no timers by design) and answer nil; live → the value (caller-owned copy).

• put_at: store + stamp; when the key list exceeds cap, evict OLDEST-INSERTED until within capacity (FIFO — the file states the LRU tradeoff the spec settled). Re-putting an existing key refreshes value + stamp without duplicating the key entry.

• Values are Text; the file says "json.encode structure into it" — no generics exist, stated, not apologized for.

• Fixture drives injected stamps: fresh hit, expiry at exactly ttl+1, eviction order under cap pressure, re-put refresh; ASan run.

• Framework standalone compile stays clean. Commit.

Task 5 — framework flags: wf_flags + cached read-through

Files:

• Create: docs/examples/writeonce-framework/store/flags.wo.

Interfaces:

• Produces: @table(name: "wf_flags") class Flag { name: Text @unique, on: Int } (Int 0/1 — Bool columns are unproven storage, spec's call) and pub class Flags { loaded: Int, cache: map<Text, Int> } with read(name, default: Bool) -> Bool and set(name, on: Bool).

• read: first call fills the map from the table (query by name — the employee-proven point-read), later calls answer from the map; absent flag → the default, uncached (so a later set is seen).

• set: update-or-insert the row, then update the map in the same call — single process, invalidation is an assignment. Durability is the table's (WAL), restart rebuilds via read.

• Framework standalone compile; behavior proven in Task 7's gate (deviation 3). Commit.

Task 6 — framework jobs: wf_jobs, enqueue, JobRunner, the idle seam

Files:

• Create: docs/examples/writeonce-framework/store/jobs.wo.
• Modify: docs/examples/writeonce-framework/http/serve.wo (Dispatcher gains fn idle(); the serve loop calls it after net.accept, BEFORE parsing the connection's first request), app.wo (App satisfies idle; jobs(take r: Jr, budget: Int) registration; Jr { r: JobRunner } wrapper, the Mw/Route pattern), README.md (the drain contract + the idle-server-drains-nothing disclosure).

Interfaces:

• Consumes: transaction { } (Tasks 1–3) only in the DEMO — enqueue itself is an ordinary insert, composition happens in app code.

• Produces: @table(name: "wf_jobs") class Job { kind: Text, payload: Text, attempts: Int, not_before: Int }; pub fn enqueue(kind, payload); pub interface JobRunner { fn run(kind: Text, payload: Text) -> Bool }; App.jobs(take r, budget); Dispatcher.idle().

• Drain (in App.idle): no runner registered → return immediately. Else query up to budget due jobs (not_before <= time.now, registration order via take), each inside try: true → delete the row; false or trap → attempts + 1 (update), row stays — retry/backoff policy is the app's (it can rewrite not_before from its own runner).

• The post-accept/pre-parse placement is the DETERMINISM the gate needs: a job enqueued by connection A never runs before A closes, and a kill after A's response provably leaves the row. Latency cost (≤ budget jobs ahead of the next request) stated in the README.

• Framework standalone compile; the serve loop's existing gates (just web-app current count) stay green with NO runner registered — the seam must cost nothing. Commit.

Task 7 — the web-app demo + the gate

Files:

• Modify: docs/examples/web-app/main.wo (transactional CreateOrder + confirm runner + GET /jobs count + POST /flags/:name + the flag-gated header on the product list), types.wo (nothing — wf_ tables come from the framework), README.md, scripts/web-app-accept.sh.

Interfaces:

• Consumes: everything above, through [deps] exactly like every other framework feature.

• CreateOrder.handle: transaction { insert Order {...}; enqueue("confirm", <order json>); } — the headline composition, one commit. A Confirm runner class answers confirm by printing the order-confirmation line to stderr and returning true; registered via app.jobs(Jr { r: Confirm {...} }, budget).

• GET /jobs (behind the existing bearer auth): pending count as JSON. POST /flags/:name: flips through Flags.set; the product list answers an extra header (e.g. x-store-banner) while the flag is on.

• Gate additions, in order: (a) POST /orders 201, then kill -9 the server IMMEDIATELY (no further requests), restart on the same WO_DATA, then GET /jobs — the confirmation line appears in the restarted server's log (the drain ran post-accept on this very request) and the count answers 0: the job survived the kill because it committed WITH the order; (b) POST /flags/banner then GET /products carries the header, restart, still carries it; (c) the standing matrix unchanged. Counts stay dynamic in the script.

• Full battery: just web-app, woc-test, oop-e2e, deps-accept, log-watcher, employee. Commit.

Task 8 — docs closeout

Files:

• Modify: docs/00-status.md (row 18 ✅ with measured results; NEXT PLAN advances to 20/21 per the order), story 18-memory-db-features.md (landing banner) then git mv into stories/.../done/ with links re-pathed and VERIFIED, docs/00-dependency-graph.md (node classes: 18 done; TPRMW unblocks), framework README.md (ledger rows: storage-integration txn-per-request now buildable; the v2 pieces ✅), compiler/src/CODE-LOGIC.md + runtime/src/CODE-LOGIC.md + database/src/CODE-LOGIC.md (the txn seams, one paragraph each).
• Apply; run just web-app once more after doc edits; commit.

Success criteria (spec, restated as the gate reads them)

1. Two inserts in one block: SIGKILL before the next request → both rows after restart (gate a); a trap unwinding out → neither row and the process keeps serving (run/transaction-abort + ASan).
2. The order's job runs after the NEXT accepted connection within budget, never before the posting connection closes, and survives a kill in between (gate a).
3. An expired or evicted cache entry answers nil with no timer having existed (run/cache-ttl, stamps injected).
4. A flipped flag holds across restart (gate b). Every standing gate green; opcode set and .wob format byte-identical.

Self-review notes

• Spec coverage: Part A semantics → Tasks 1–3 (observable rules mapped one-to-one; the early-exit hole closed by deviation 1); cache → T4; flags → T5; jobs + seam → T6; demo + gate → T7; out-of-scope list untouched. Corpus/gate split follows deviations 2–3.
• Type consistency: the three engine entry points, the two E-codes (E110/E112), wf_flags/wf_jobs, get_at/put_at, JobRunner.run(kind, payload) -> Bool, App.jobs(take r, budget), Dispatcher.idle() — spelled identically in every task that names them.
• Risk, disclosed: the abort walk's index restoration is the one place correctness is subtle (indexed column updated then aborted); Task 1's unit test pins exactly that case before any VM work stacks on it.
• Ordering: engine (T1) before VM (T3) with the language (T2) between so T3 has both; cache/flags (T4/T5) are independent and could land any time, kept after the critical path so the risky work gets the freshest attention.