shoney/writeonce
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

docs: pending iterations renumbered by dependency + priority

Browse source 
- developer directive: pending iteration IDs now ARE the priority order;
  LANDED iterations keep historical numbers (code comments and commit
  history cite them — records, not a queue); 8/11 (the half-landed
  arc), 17 (parked, artifacts on a branch), 18 (next, artifacts named)
  also frozen
- mapping (recorded in 00-story): 19<-20 Float+Bytes, 20<-9c attach,
  21<-9d keypair, 22<-9e benchmarks, 23<-9f io_uring WAL, 24<-19 chat,
  25<-10 services, 26<-12 blue-green, 27<-9g query corpus,
  28<-14 skillhost, 29<-13 metaprogramming
- 11 story files renamed; every doc reference re-numbered (word-boundary
  sweep for the lettered 9x ids, phrase-level for numeric ones); the
  iterations table rewritten with Seq == priority and "(was N)" notes;
  story-scoped link check: zero broken
- merge-recovery folded in: the partial master merge had dropped the
  chat story, the fibers exploration note, the arc spec+plan, the
  framework-v2 plan, and the iteration-17 spec+plan — all restored from
  their branches and renumbered consistently

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
shoney.arickathil 2026-08-20 14:31:09 +02:00
parent 4c8a3bd2ed
commit 5521d21a84
31 changed files with 1626 additions and 147 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/00-code-review.md
View file

@ -12,4 +12,4 @@ Native speed — the big one. Everything is interpreted: ~40× behind Go on raw
- Language expressiveness. No generics — the cache stores Text and tells you to json.encode; no function values or closures (doctrine, but it's why every handler is a class with one method); byte-based strings with no Unicode awareness; no Result-style error values (traps + try only); pattern matching is a switch, not destructuring. Some of this is deliberate rejection, but "deliberate" doesn't make the expressiveness appear.
- Concurrency holes the arc hasn't closed. send is one-way — no reply/request-response primitive (my own benchmarks couldn't await the actor and had to sleep); no supervision, links, or actor death (actors live until process end); unbounded mailboxes with zero backpressure; no timers beyond sleep; round-robin placement with no work stealing; multi-shard DB access still traps (stage 3 unbuilt); accept lives on one shard.
- Production plumbing. No TLS anywhere (proxy-mandated forever), no HTTP/2 or WebSockets yet, no crypto primitives (blocked on the bit-ops-vs-builtins fork), observability is print/stderr — no metrics, tracing, or profiler; no debugger, no LSP (discussed, never built); deps are git-rev-only with no registry, no transitive resolution, no semver; blue-green deploy and schema migrations are recorded futures, not features.
- Proof maturity. 9e's benchmark battery has never run — every number so far is a scratch measurement on one machine; TSan covers one demo; no fuzzing, no CI beyond local just, and the whole ecosystem is one framework, five samples, and one committed consumer. The honest summary: the architecture is ahead of the product — the doctrine bets (ownership+inference, actors, one binary, io_uring) are landing and measurable, while the surface a developer touches daily (types, tooling, ecosystem) is years behind the languages it benchmarks against.
- Proof maturity. 22's benchmark battery has never run — every number so far is a scratch measurement on one machine; TSan covers one demo; no fuzzing, no CI beyond local just, and the whole ecosystem is one framework, five samples, and one committed consumer. The honest summary: the architecture is ahead of the product — the doctrine bets (ownership+inference, actors, one binary, io_uring) are landing and measurable, while the surface a developer touches daily (types, tooling, ecosystem) is years behind the languages it benchmarks against.

32
docs/00-dependency-graph.md
View file

@ -7,7 +7,7 @@
> green is startable today. Rebuilt 2026-08-20 from a sweep of every
> story/spec/plan markdown (the "misses" pass: iteration 17's outgoing
> edges, the concurrency chain, the post-12 parked drain, 9b→10,
> 14's gap fan-out, 9c's fiber caveat).
> 14's gap fan-out, 20's fiber caveat).
## 1. Story iterations
@ -30,17 +30,17 @@ flowchart TD
FWREORG["framework internal/ reorg + check mode (kills the --emit workaround; WO-E108/E109 reserved)"]:::parked
I18["18 framework v2: transaction{} + cache/flags/jobs (spec APPROVED — the next implementation)"]:::specd
I9c["9c cross-program tables (half-built)"]:::open
I9d["9d keypair attach auth (half-built; crypto+handshake already on its branch)"]:::open
I9e["9e durability + throughput baseline"]:::open
I9c["20 cross-program tables (half-built)"]:::open
I9d["21 keypair attach auth (half-built; crypto+handshake already on its branch)"]:::open
I9e["22 durability + throughput baseline"]:::open
I8["8 shard-actor runtime"]:::open
I9f["9f io_uring group-commit"]:::open
I9f["23 io_uring group-commit"]:::open
I10["10 HTTP service layer (lowers onto the framework)"]:::open
I11["11 fibers"]:::open
I12["12 blue-green deploy"]:::open
I13["13 metaprogramming @derive"]:::open
I14["14 skillhost workload (demoted)"]:::open
I9g["9g query grammar corpus (likely collapses)"]:::open
I9g["27 query grammar corpus (likely collapses)"]:::open
GAPS["14's gap fan-out: bounded subprocess, stdin/stdout transport, fs metadata, FFI-vs-out-of-process"]:::open
DRAIN["post-12 parked drain: pub(read)/using/#if, WO-E225, ADT roster, group-by"]:::parked
@ -75,14 +75,14 @@ flowchart TD
```
Reading it: **18 is the only spec-approved open node with all
prerequisites green — the next implementation.** After 18: 9c/9d and 9e
are startable (chosen order: 9c/9d first — half-built branches rot).
prerequisites green — the next implementation.** After 18: 20/21 and 22
are startable (chosen order: 20/21 first — half-built branches rot).
17 unparks on directive: its prerequisites landed, its spec+plan wait on
branch `library-internal`, and its landing brings the framework reorg
node with it. 13 and the parked drain sit behind 12 by the 2026-08-08
scope directive (dashed), not by any technical edge.
## 2. The concurrency chain (iterations 8 / 9f / 11 and everything they gate)
## 2. The concurrency chain (iterations 8 / 23 / 11 and everything they gate)
The runtime's concurrency work is the single biggest unlocker — every
⏸ row in the framework ledger and two v2 follow-ons hang off it.
@ -95,17 +95,17 @@ flowchart TD
I7b2["7b per-shard collector (done — the precondition 8 waited on)"]:::rt
I8x["8 shard-actor runtime: thread-per-core, ownership-move messages"]:::rt
I9fx["9f io_uring group-commit (batch = the shard tick)"]:::rt
I9fx["23 io_uring group-commit (batch = the shard tick)"]:::rt
I11x["11 fibers: reduction-budget preemption, blocking builtins park"]:::rt
I9ex["9e baseline (numbers 8/9f sign against)"]:::rt
I9ex["22 baseline (numbers 8/23 sign against)"]:::rt
KEEPAL["keep-alive parking retired (close-when-idle policy dies; parked fds)"]:::gated
H2C2["h2c HTTP/2 cleartext (spec §C: also needs 9f)"]:::gated
H2C2["h2c HTTP/2 cleartext (spec §C: also needs 23)"]:::gated
STREAM2["request body streaming + backpressure"]:::gated
SRESP2["streaming responses + explicit commit point"]:::gated
CANCEL2["per-request cancellation propagation"]:::gated
PUBSUB2["pub/sub + WebSockets (rejected until here)"]:::gated
ASYNC9C["9c async attach statements (rejected-for-now alternative)"]:::gated
ASYNC9C["20 async attach statements (rejected-for-now alternative)"]:::gated
TIMEOUTS2["idle timeouts become schedulable (net seam still needed)"]:::gated
FIBJOBS2["fiber-scheduled jobs (replaces drain-on-request; queue table stays)"]:::v2
@ -164,7 +164,7 @@ flowchart TD
UNIX["unix socket binding"]:::blocked
PEERV["trusted-proxy PEER verification"]:::blocked
CRYPTO["GATE: crypto fork — C builtins vs language bit ops (brainstorm); digests want iteration 20's Bytes"]:::gate
CRYPTO["GATE: crypto fork — C builtins vs language bit ops (brainstorm); digests want iteration 19's Bytes"]:::gate
SHA["SHA-256/512, HMAC, CRC32"]:::blocked
ETAG["ETag + conditional requests"]:::blocked
COOKIE["signed cookies"]:::blocked
@ -174,7 +174,7 @@ flowchart TD
JWT["JWT HS256 (HARD STOP after)"]:::blocked
RADIX["radix-tree routing"]:::blocked
I9E3["GATE: 9e measures the linear scan"]:::gate
I9E3["GATE: 22 measures the linear scan"]:::gate
STORAGE["storage-integration rows: migrations (future story), eager loading + tenant roots (query-surface work, 9-series)"]:::blocked
@ -195,7 +195,7 @@ flowchart TD
Green nodes (CORS, security headers, host validation, strict-parsing
audit, wildcards, route groups, `req.ctx`, XFF parsing, Accept
negotiation) need nothing — startable in any order, gated by
`just web-app`. Note: 9d's keypair crypto is its own C implementation
`just web-app`. Note: 21's keypair crypto is its own C implementation
(already on branch `keypair-auth`) — it neither waits for nor feeds the
crypto-fork gate.

70
docs/00-status.md
View file

@ -68,7 +68,7 @@ operators, streaming/cancellation park behind 8/11). The memory-rich
features are **framework v2** = iteration 18 (spec APPROVED 2026-08-20,
plan next): TTL cache, @table flags, durable job queue with
drain-on-request, `transaction { }` over the WAL's staged batch. After 18,
the order resumes at 9c/9d. Edges: [00-dependency-graph.md](00-dependency-graph.md).
the order resumes at 20/21. Edges: [00-dependency-graph.md](00-dependency-graph.md).
---
@ -171,18 +171,18 @@ that sequences its tasks. Read one, approve, then the next starts.
| 8 | [Shard-actor runtime](stories/language-runtime-database/08-shard-actor-runtime.md) | ⬜ |
| 9 | [Database engine](stories/language-runtime-database/done/09-database-engine.md) | 🔄 engine complete (storage/WAL/indexes/insert-update-delete); reads land with 9b |
| 9b | [`@table`, relations, query](stories/language-runtime-database/done/09b-table-relations-query.md) | 🔄 query surface + relations + FK done (branch query-surface); group-by parked |
| 9c | [Cross-program tables](stories/language-runtime-database/refine/09c-cross-program-tables.md) | 🔄 channel done (branch ipc-attach); manifest+binding pending |
| 9d | [Keypair attach auth](stories/language-runtime-database/refine/09d-keypair-attach-auth.md) | 🔄 crypto+handshake done (branch keypair-auth); manifest pending |
| 9e | [Durability, throughput, scale](stories/language-runtime-database/refine/09e-durability-throughput-scale.md) | ⬜ needs a spec first |
| 9f | [io_uring group-commit](stories/language-runtime-database/refine/09f-io-uring-commit.md) | ⬜ after 8 + 9e |
| 9g | [Query grammar corpus](stories/language-runtime-database/refine/09g-query-grammar-corpus.md) | ⬜ needs a spec first |
| 10 | [HTTP service layer](stories/language-runtime-database/10-http-service.md) | ⬜ | Hold |
| 20 | [Cross-program tables](stories/language-runtime-database/refine/20-cross-program-tables.md) | 🔄 channel done (branch ipc-attach); manifest+binding pending |
| 21 | [Keypair attach auth](stories/language-runtime-database/refine/21-keypair-attach-auth.md) | 🔄 crypto+handshake done (branch keypair-auth); manifest pending |
| 22 | [Durability, throughput, scale](stories/language-runtime-database/refine/22-durability-throughput-scale.md) | ⬜ needs a spec first |
| 23 | [io_uring group-commit](stories/language-runtime-database/refine/23-io-uring-commit.md) | ⬜ after 8 + 22 |
| 27 | [Query grammar corpus](stories/language-runtime-database/refine/27-query-grammar-corpus.md) | ⬜ needs a spec first |
| 10 | [HTTP service layer](stories/language-runtime-database/25-http-service.md) | ⬜ | Hold |
| 11 | [Fibers](stories/language-runtime-database/refine/11-fibers.md) | ⬜ | Hold |
| 12 | [Blue-green deploy](stories/language-runtime-database/12-blue-green-deploy.md) | ⬜ | Hold |
| 13 | [Compile-time metaprogramming](stories/language-runtime-database/refine/13-compile-time-metaprogramming.md) | ⬜ needs a spec first |
| 14 | [skillhost host workload](stories/language-runtime-database/refine/14-skillhost-host-workload.md) | ⬜ gaps recorded (branch query-grammar found skillhost needs no new query grammar); each gap a candidate iteration |
| 12 | [Blue-green deploy](stories/language-runtime-database/26-blue-green-deploy.md) | ⬜ | Hold |
| 13 | [Compile-time metaprogramming](stories/language-runtime-database/refine/29-compile-time-metaprogramming.md) | ⬜ needs a spec first |
| 14 | [skillhost host workload](stories/language-runtime-database/refine/28-skillhost-host-workload.md) | ⬜ gaps recorded (branch query-grammar found skillhost needs no new query grammar); each gap a candidate iteration |
| 15 | [deps: `wo.toml [deps]`](stories/language-runtime-database/done/15-deps-package-manager.md) | ✅ **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 |
| 16 | [web framework](stories/language-runtime-database/done/16-web-framework.md) | ✅ **landed 2026-08-19** — writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (§C) behind 8/9f/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 |
| 16 | [web framework](stories/language-runtime-database/done/16-web-framework.md) | ✅ **landed 2026-08-19** — writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (§C) behind 8/23/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 |
| 17 | [library projects + `internal/`](stories/language-runtime-database/17-library-projects-internal.md) | ⏸ **PARKED 2026-08-20** (developer directive; framework v1 first) — forks settled, spec + plan approved and ready on branch `library-internal`: kind = "library" key; Go internal/ rule, dep-boundary-only; lib+bin dual; VM/GC untouched by design |
| 18 | [framework v2: memory-rich features](stories/language-runtime-database/18-memory-db-features.md) | 🔄 **spec APPROVED 2026-08-20, plan next** ([spec](superpowers/specs/2026-08-20-memory-db-features-design.md)): TTL cache + @table flags + durable job queue (drain-on-request) + `transaction { }` over the WAL's staged batch; pub/sub REJECTED until 8/11 |
@ -192,7 +192,7 @@ that sequences its tasks. Read one, approve, then the next starts.
| Track | Item | Where |
| -------- | --------------------------------------------------------------------------- | ---------------------------------------------------------- |
| Language | nothing active — the framework v1-polish slice landed 2026-08-20 (branch framework-v1, awaiting merge); next per the order: brainstorm 9c/9d's forks | [order](#implementation-order-re-sequenced-2026-08-20--framework-goal) |
| Language | nothing active — the framework v1-polish slice landed 2026-08-20 (branch framework-v1, awaiting merge); next per the order: brainstorm 20/21's forks | [order](#implementation-order-re-sequenced-2026-08-20--framework-goal) |
Off-goal work is parked; the goal (2026-08-20) is the web framework as a
polished micro-framework v1 — iteration 17 (library kind + `internal/`) is
@ -362,11 +362,11 @@ The C proving-ground work (`exploration/c-runtime/`, phases A–F: 859k reads/s,
### Implementation order (re-sequenced 2026-08-20 — framework goal)
The goal is the web framework as a first-class library, so the framework
line leads and the workload-driven extras (14, 9g) demote behind it.
Dependency rules that force the shape: 9f explicitly after 8 + 9e; 9c
"precedes iteration 10"; 9d's plan folds into 9c's; 12 only after 9 + 10
line leads and the workload-driven extras (14, 27) demote behind it.
Dependency rules that force the shape: 23 explicitly after 8 + 22; 20
"precedes iteration 25"; 21's plan folds into 20's; 12 only after 9 + 10
(catalog to diff, HTTP to build on); 11 rides 8's shard scheduler; h2c
parked behind 8/9f/11; the post-12 parked list stays parked by the
parked behind 8/23/11; the post-12 parked list stays parked by the
2026-08-08 scope directive. (Iteration 7 dropped from this list — landed
2026-08-15.)
@ -379,27 +379,27 @@ parked behind 8/9f/11; the post-12 parked list stays parked by the
APPROVED 2026-08-20, the only all-green spec-approved node in the
[dependency graph](00-dependency-graph.md) — plan next, then
implement.
3. **9c then 9d** — finish the half-done branches (ipc-attach: manifest +
binding; keypair-auth: manifest) before they rot; 9d folds into 9c's
3. **20 then 21** — finish the half-done branches (ipc-attach: manifest +
binding; keypair-auth: manifest) before they rot; 21 folds into 20's
plan; both must precede 10.
4. **9e** — the measurement backbone; baseline single-shard BEFORE the
runtime restructure so 8/9f/11 sign against real numbers.
4. **22** — the measurement backbone; baseline single-shard BEFORE the
runtime restructure so 8/23/11 sign against real numbers.
5. **8** — shard-actor; the framework's multi-core serving story; unblocks
9f, 11, h2c.
6. **9f** — io_uring group-commit; explicitly after 8 + 9e.
23, 11, h2c.
6. **23** — io_uring group-commit; explicitly after 8 + 22.
7. **11** — fibers; retires the framework's disclosed keep-alive limit (an
idle connection starving accept forced close-when-idle in iteration 16 —
parked fds fix it properly); with 8 + 9f done, **h2c unparks** (spec §C)
parked fds fix it properly); with 8 + 23 done, **h2c unparks** (spec §C)
as the framework's HTTP/2 slice.
8. **10** — HTTP service layer; after 9c by its own precedence note;
8. **10** — HTTP service layer; after 20 by its own precedence note;
`service` blocks lower onto the framework instead of a parallel stack.
9. **12** — blue-green; prerequisites 9 + 10 now exist; completes the
framework's deploy story.
10. **9g then 14** — demoted with the goal shift: skillhost is no longer the
driving workload; 9g likely collapses to "confirm `len(query)` + add
`exists`" and precedes 14 when they run.
10. **27 then 14** — demoted with the goal shift: skillhost (28) is no longer the
driving workload; 27 likely collapses to "confirm `len(query)` + add
`exists`" and precedes 28 when they run.
11. **13 + parked drain** — metaprogramming (spec first), then group-by,
`pub(read)`/`using`/`#if`, ADT roster, WO-E225 — held behind 12 by the
`pub(read)`/`using`/`#if`, ADT roster, WO-E225 — held behind 26 by the
scope directive.
### Language track — sequenced, on the critical path
@ -412,18 +412,18 @@ parked behind 8/9f/11; the post-12 parked list stays parked by the
| 8 | Shard-actor runtime | [plan 4](superpowers/plans/2026-08-01-shard-actor-vm-runtime.md) |
| 9 | Database engine binding | [plan 5](superpowers/plans/2026-08-01-db-engine-binding.md) |
| 9b | `@table` + relations + language-integrated query — comprehension queries, `ref`/`backlink` navigation, GroupBy aggregates; acceptance: new `docs/examples/employee` sample | [spec](superpowers/specs/2026-08-15-table-relations-query-design.md) · [plan](plan/compiler/2026-08-15-employee-relations-query.md) |
| 9c | Cross-program tables — attach to a running program's database (IPC string in wo.toml, manifest-granted rights, owner stays the single writer) | **no spec yet** — four open forks recorded in the iteration; brainstorm before planning |
| 9d | Keypair attach auth — mutual challenge–response, grants name public keys, uid superseded | **no spec yet** — four forks recorded; plan folds into 9c's |
| 9e | Durability + throughput + scale — restart-persistence, read/write benchmark, ~1M rows; the gate every later optimization re-runs | **no spec yet** — four forks recorded; the measurement backbone |
| 9f | io_uring group-commit write path — batched durability overlapped on shard threads, fsync fallback | **no spec yet** — brainstorm after iterations 8 + 9e |
| 9g | Query grammar from real embedded-DB corpora — whole-query count + correlated exists, driven by the skillhost SQL catalogue; add only what a corpus uses | **no spec yet** — three forks; may collapse to "confirm len(query) + add exists" |
| 20 | Cross-program tables — attach to a running program's database (IPC string in wo.toml, manifest-granted rights, owner stays the single writer) | **no spec yet** — four open forks recorded in the iteration; brainstorm before planning |
| 21 | Keypair attach auth — mutual challenge–response, grants name public keys, uid superseded | **no spec yet** — four forks recorded; plan folds into 20's |
| 22 | Durability + throughput + scale — restart-persistence, read/write benchmark, ~1M rows; the gate every later optimization re-runs | **no spec yet** — four forks recorded; the measurement backbone |
| 23 | io_uring group-commit write path — batched durability overlapped on shard threads, fsync fallback | **no spec yet** — brainstorm after iterations 8 + 22 |
| 27 | Query grammar from real embedded-DB corpora — whole-query count + correlated exists, driven by the skillhost SQL catalogue; add only what a corpus uses | **no spec yet** — three forks; may collapse to "confirm len(query) + add exists" |
| 14 | skillhost host workload — port skillhost (MCP host + confined script runner) to writeonce; drives the missing host capabilities into the open (bounded subprocess, stdin/stdout transport, fs metadata, FFI-vs-out-of-process) | **no spec yet** — gaps recorded in the iteration; each gap brainstormed on demand, bounded-subprocess first |
| 17 | library projects + dependency privacy — `wo.toml` kind = "library" (checkable without entry, dual lib+bin) + Go-style `internal/` at the [deps] boundary; framework reorg demonstrates both | **forks settled 2026-08-20** — decisions + framework/compiler/VM/GC impact in the iteration; spec/plan next |
| 10 | HTTP service layer | [plan 6](superpowers/plans/2026-08-01-http-service-layer.md) |
| 11 | Fibers | vision §3, [blue-green exploration](plan/exploration/blue-green-vm/00-vision.md) |
| 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9–10 |
| 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9 + 25 |
### Language track — parked until after iteration 12
### Language track — parked until after iteration 26
Recorded 2026-08-08 by scope directive; nothing here lands before the
log-watcher proof.

6
docs/examples/employee-list/README.md
View file

@ -2,8 +2,8 @@
> **Status: target workload — does not compile on today's toolchain.**
> Written ahead of iterations
> [9c (cross-program tables)](../../stories/language-runtime-database/refine/09c-cross-program-tables.md)
> and [9d (keypair attach auth)](../../stories/language-runtime-database/refine/09d-keypair-attach-auth.md),
> [20 (cross-program tables)](../../stories/language-runtime-database/refine/20-cross-program-tables.md)
> and [21 (keypair attach auth)](../../stories/language-runtime-database/refine/21-keypair-attach-auth.md),
> the way every acceptance sample here precedes its features. It also leans
> on 9/9b (the [employee sample](../employee/) it attaches to must run
> first).
@ -37,6 +37,6 @@ the source says `employee.Employee`.
| `employee-list staff <dept>` | unique-name index probe + `staff` backlink scan, both in A |
| `employee-list probe-write` | the rights matrix: registered read-only, so the insert traps with access-denied (caught, `DENIED …`, exit 4) and A's row count is unchanged |
The 9d acceptance drives the rest from the outside: wrong key, no key,
The 21 acceptance drives the rest from the outside: wrong key, no key,
same-uid-wrong-key, impostor socket, handshake replay, key rotation — see
the iteration's criteria; this sample is the workload they run against.

2
docs/examples/writeonce-framework/README.md
View file

@ -87,7 +87,7 @@ first (pure `.wo` cannot express it yet).
| Item | State |
| --- | --- |
| Path matching | 🔶 linear scan, first-match-wins; a radix tree is a performance slice that waits for iteration 9e to MEASURE it first |
| Path matching | 🔶 linear scan, first-match-wins; a radix tree is a performance slice that waits for iteration 22 to MEASURE it first |
| Method dispatch · path params · 404 · 405+`Allow` | ✅ |
| Wildcards | ⬜ only `:param` today; `*rest` capture is a candidate slice |
| Precedence rules | 🔶 registration order IS the rule (documented); specificity-based precedence unneeded until wildcards exist |

159
docs/plan/exploration/fibers/00-fibers.md Normal file
View file

@ -0,0 +1,159 @@
# Fibers — the concurrency model, its terminology, and why writeonce chose what it chose
> Exploration/reference note (no status banner by board convention).
> The normative decisions live in the arc spec
> ([`2026-08-20-shard-fiber-arc-design.md`](../../../superpowers/specs/2026-08-20-shard-fiber-arc-design.md))
> and iterations [8](../../../stories/language-runtime-database/refine/08-shard-actor-runtime.md) /
> [11](../../../stories/language-runtime-database/refine/11-fibers.md); this
> page explains the WHY at doctrine depth. Written 2026-08-20, when this
> file was also the target of a dangling reference from iteration 11 —
> it exists now.
> **Demonstrated live** by [`docs/examples/fibers`](../../../examples/fibers/README.md)
> (`just fibers`, 8 checks): byte-exact budget interleave, a parked
> sleeper blocking nobody, on the io_uring AND epoll backends.
## What a green thread is, and why anyone bothers
A green thread (fiber, virtual thread, coroutine — the tradition picks
the name) is a thread scheduled by the LANGUAGE RUNTIME, not the kernel.
The kernel sees a handful of OS threads; the runtime multiplexes
thousands-to-millions of logical threads on top.
The economics: an OS thread costs a fixed stack reservation (typically
1–8 MiB of address space) and every context switch goes through the
kernel scheduler (CFS is O(log N), and task counts in the hundreds
already hurt). A green thread can start around 2 KiB and switch in tens
of nanoseconds — a register save and a pointer swap, no syscall. That is
the difference between a few thousand concurrent connections and a few
million.
## The terminology, untangled
- **Green thread** — originally Java pre-1.3's model, which was M:1
(everything on ONE OS thread — no parallelism at all). Now used
loosely for any user-space thread.
- **Fiber** — usually implies COOPERATIVE: it runs until it explicitly
yields.
- **Coroutine** — the broadest term; covers stackful and stackless both.
- **Goroutine** — Go's M:N implementation, preemptively scheduled.
- **Virtual thread** — Java 21's, also M:N.
The load-bearing distinction between "fiber" and "goroutine" is NOT the
stack — it is the EVICTION POLICY. A classic fiber owns the CPU until it
yields; one selfish loop starves everything behind it. A goroutine can
be interrupted WITHOUT ITS CONSENT (since Go 1.14, via a signal that
stops it at a safe instruction).
M:1 vs M:N matters concretely here: one process embedding its own
storage would leave every core but one idle under M:1. writeonce is M:N
— stage 2 of the arc pins one kernel thread per core (shards), and
fibers multiplex above them.
## writeonce's model: cooperative mechanics, accounting-based eviction
The BEAM shape. Mechanically cooperative — no signals, no interrupts,
no async preemption machinery — but every loop back-edge decrements a
REDUCTION BUDGET (`WO_REDUCTIONS`, default 4000), and at zero the fiber
is re-queued whether it likes it or not.
What that buys, all three at once:
- goroutine-grade starvation-freedom (the corpus pins budget-1 as EXACT
round-robin across fibers — `runtime/test/test_fiber.c`);
- fiber-grade DETERMINISM: same program, same schedule, byte-identical
output — signal-based preemption points are nondeterministic by
nature, Go's included;
- zero interruption machinery: an interpreter's dispatch loop is always
at a safe point, so the "where can I safely stop this thread"
problem Go solves with signal handlers and pc maps costs nothing.
One implementation lesson worth keeping (it cost a livelock): the
budget check runs at loop back-edges AFTER the jump lands, so the saved
resume pc is the loop head. A pre-instruction save at budget 1
re-executes the jump straight into the same decrement, forever.
## Should the grammar have `async`? No — permanently rejected
`async` is only NECESSARY in stackless designs, where the compiler must
know statically which functions can suspend so it can transform each
into a state-machine struct. That requirement is function COLORING: an
async function can only be called from another async function, so the
keyword propagates virally up every call chain, and the ecosystem forks
into sync and async flavors of everything. In writeonce it would color
the query API, then every handler that touches data — the exact code
that most wants to read as straight-line logic.
With stackful fibers, a blocking read ten frames deep just parks. The
suspension point is a RUNTIME fact, not a TYPE-SYSTEM fact. Same source
text in program mode and server mode; no keyword; no coloring. This is
iteration 11's "async/await — permanently rejected surface, not
deferred", with the reasoning attached.
## Stackful vs stackless: the real fork, and the recommendation
**Stackful** (goroutines, Java virtual threads, classic fibers): each
logical thread owns a real, growable stack; it can suspend from ANY
depth. Costs: growth-by-copy must find and rewrite every pointer into
the old stack (Go's machinery), and FFI is the classic killer — C
callees assume a large contiguous stack and know nothing about growth,
so every foreign call needs a stack switch or a guaranteed segment.
**Stackless** (Rust futures, JS promises, C# async, Python coroutines):
the compiler rewrites each async function into a fixed-size state
object. Cheaper, runtime-optional — and it buys the coloring above,
plus suspension only at explicit await points.
**Recommendation, and what is already built: STACKFUL — because both of
its classic costs vanish in this stack.**
1. **No FFI, by doctrine.** The main argument against small growable
stacks is C interop. writeonce has no C to call (the reject row is
load-bearing). The cost is simply absent.
2. **The "stack" is register-indexed, not address-based.** A
`wo_fiber`'s state is the interpreter's register window + frame
array — frames reference registers by INDEX, so a context is
relocatable by construction. Growth is allocate-larger + memcpy with
ZERO pointer fixups; Go's hardest stackful problem does not exist
here. Contexts are ~42 KiB fixed today; start-small (~4 KiB)
growable contexts are the recorded improvement that takes fiber
counts from thousands toward millions (arc spec, "Fiber context
growth").
## The two problems every design must solve, and where they land here
- **Blocking syscalls.** A green thread calling fsync blocks the OS
thread carrying it and strands every fiber behind it — and an
embedded database fsyncs on every commit, milliseconds each, an
eternity for a scheduler. writeonce's answer is io_uring, twice: the
arc's per-shard ring parks blocking `net`/`time` builtins (T4,
readiness ops first, then reads/writes as ring ops), and iteration
23 rides the SAME ring for WAL WRITE+FSYNC group-commit so the
DB-owner shard keeps executing while durability drains. epoll
survives only as the portability fallback behind a startup probe
(`WO_IO=uring|epoll`) — seccomp'd containers routinely deny io_uring,
and the binary must run everywhere. The reference cards live in
[`../linux/07-io_uring.md`](../linux/07-io_uring.md).
- **Preemption.** Solved by the reduction budget above. And to the
standing question "does preemption depend on FFI?" — NO, inversely:
preemption holes come from frames the runtime cannot interrupt, which
in Go means foreign C frames (its signal preemption skips them). No
FFI means no foreign frames; the only non-preemptible regions are our
OWN builtins — C we control, bounded, and the blocking ones become
parked ring ops. FFI would have CREATED the dependency; its absence
is why the budget is airtight.
## Precedent survey (what was adopted, what was rejected)
- **BEAM (Erlang/Elixir): adopted** — reduction-budget preemption,
actor mailboxes, one-message-at-a-time delivery, isolated failure
(a fiber's uncaught trap kills that fiber alone).
- **Go: half-adopted** — M:N over pinned threads yes; stack-copy
pointer rewriting unnecessary here (register-indexed contexts);
signal-based async preemption rejected (nondeterministic, and the
budget makes it redundant).
- **Rust/Tokio, JS, C#, Python: rejected** — stackless coloring is the
cost this language exists to not pay.
- **Java Loom: matches** — park-under-a-blocking-API is exactly the
stdlib posture (`net.read` blocks in program mode, parks under
fibers, same signature).

50
docs/stories/language-runtime-database/00-story.md
View file

@ -40,7 +40,12 @@ iterations); no commits by agents — drafts go to `.dev/commit.md`.
## Iterations (rows in dependency order — `#` is an immutable ID, not a rank)
Re-sequenced 2026-08-20 from the edges in
RENUMBERED 2026-08-20 (developer directive): pending iterations carry
fresh IDs in priority order; LANDED iterations keep their historical
numbers (code comments and commit history cite them — records, not a
queue). Mapping: 19←20(scalars), 20←9c, 21←9d, 22←9e, 23←9f, 24←19(chat),
25←10, 26←12, 27←9g, 28←14, 29←13; 8, 11, 17, 18 unchanged.
Re-sequenced from the edges in
[`docs/00-dependency-graph.md`](../../00-dependency-graph.md): landed
rows in landing order, then the pending rows in implementation order.
File names keep their IDs — every board, spec, and plan references
@ -52,28 +57,29 @@ iterations by number, so numbers never renumber.
| 2 | 2 | [VM core](done/02-vm-core.md) | `wovm`: `.wob` loader, register interpreter, arena, borrow word, `@gc` collector |
| 3 | 3 | [Compiler front](done/03-compiler-front.md) | `woc`: lexer → parser → typechecker → ownership pass, diagnostics |
| 4 | 4 | [Single binary end-to-end](done/04-single-binary-e2e.md) | emitter + conformance corpus + `woc build` self-contained binary |
| 5 | 5 | [Language surface](05-language-surface.md) | Haxe-parity adoptions: switch, records, optionals, try/catch, statics, modules… (`pub(read)`/`using`/`#if` remainders sit in the post-12 drain) |
| 5 | 5 | [Language surface](05-language-surface.md) | Haxe-parity adoptions, grammar + strictness halves (landed in waves through 2026-08-20) |
| 6 | 6 | [Program mode + stdlib](done/06-program-mode-stdlib.md) | `fn main`, exit codes, `fs`/`proc`/`net`/`time`/`json` builtins |
| 7 | 7 | [log-watcher proof](done/07-logwatcher-proof.md) | the driving workload compiled, executable, soak-proven (landed 2026-08-15) |
| 8 | 7b | [Inferred GC + mark-sweep](done/07b-inferred-gc-mark-sweep.md) | `@gc` removed from the language; compiler infers GC-ness; RC replaced by incremental per-shard tri-color mark-sweep |
| 8 | 7b | [Inferred GC + mark-sweep](done/07b-inferred-gc-mark-sweep.md) | `@gc` removed; GC-ness inferred; RC replaced by incremental per-shard tri-color mark-sweep |
| 9 | 9 | [Database engine](done/09-database-engine.md) | class-shaped tables, typed WAL + recovery, `insert`/`select` execute |
| 10 | 9b | [`@table`, relations, query](done/09b-table-relations-query.md) | `@table` becomes real storage; typed `ref`/`backlink`/`multi` relations; compiler-checked LINQ-shaped queries lowered to engine ops |
| 11 | 15 | [deps: `wo.toml [deps]`](done/15-deps-package-manager.md) | exact-rev git dependencies + `wo.lock` + `.wo-deps` cache; `use <dep>` resolves a fetched project as a module root; flat-only, network-free when locked |
| 12 | 16 | [web framework](done/16-web-framework.md) | a `.wo`-library framework (HTTP/1.1 keep-alive behind a TLS-terminating proxy): router, `Handler`/`Middleware` interfaces, auth, all three body hooks, `@table` data layer; `docs/examples/web-app` consumes it via `[deps]`; h2c parked behind 8/9f/11 |
| **13** | **18** | [framework v2: memory-rich features](18-memory-db-features.md) | **NEXT — spec approved 2026-08-20**: TTL cache, @table feature flags with cached reads, durable @table job queue with drain-on-request, `transaction { }` exposing the WAL's staged batch (enqueue + write, one commit — no outbox) |
| 14 | 9c | [Cross-program tables](refine/09c-cross-program-tables.md) | attach to a running program's database over a local IPC channel: manifest-granted rights, typed statements, owner stays the single writer (channel half-built) |
| 15 | 9d | [Keypair attach auth](refine/09d-keypair-attach-auth.md) | program identity is a keypair: mutual challenge–response at attach, grants name public keys, replay-proof (crypto half-built; plan folds into 9c's) |
| 16 | 9e | [Durability, throughput, scale](refine/09e-durability-throughput-scale.md) | restart-persistence proof, read/write benchmark, ~1M-row load; the baseline 8/9f/11 sign against |
| 17 | 8 | [Shard-actor runtime](08-shard-actor-runtime.md) | thread-per-core shards, per-shard heaps, ownership-move messaging (collector precondition met by 7b) |
| 18 | 9f | [io_uring group-commit](refine/09f-io-uring-commit.md) | replace fsync-per-commit with io_uring batched durability on the shard tick; fsync fallback kept (after 8 + 9e, explicit) |
| 19 | 11 | [Fibers](refine/11-fibers.md) | green threads on the shard scheduler: reduction-budget preemption, blocking builtins park; unparks h2c (with 8/9f), streaming, cancellation, pub/sub, fiber jobs |
| 20 | 10 | [HTTP service layer](10-http-service.md) | `service` blocks lower onto the framework (after 9b + 9c by their own precedence notes) |
| 21 | 12 | [Blue-green deploy](12-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback (plan authored after 9 + 10) |
| 22 | 9g | [Query grammar corpus](refine/09g-query-grammar-corpus.md) | grow the query grammar from real corpora; likely collapses to "confirm `len(query)` + add `exists`"; precedes 14 |
| 23 | 14 | [skillhost host workload](refine/14-skillhost-host-workload.md) | host-shaped driving workload naming runtime gaps (bounded subprocess, stdin/stdout transport, fs metadata, FFI-vs-out-of-process) — demoted with the framework goal |
| 24 | 13 | [Compile-time metaprogramming](refine/13-compile-time-metaprogramming.md) | `@derive(Json/Csv/Eq/Hash/Show)` from class-table metadata; held behind 12 with the parked drain by the 2026-08-08 scope directive |
| 26 | 20 | [Float + Bytes](20-missing-scalar-types.md) | the missing scalars, full stack: IEEE-quiet f64 through literals/VM/@table/WAL/json (fractions decode at last) + Bytes as the binary carrier; forks settled 2026-08-20, spec next — feeds 19 (WS frames) and the crypto fork (digests) |
| ⏸ | 17 | [library projects + `internal/`](17-library-projects-internal.md) | **PARKED** (spec + plan approved, branch `library-internal`) — `wo.toml` kind = "library" + Go's `internal/` rule; slots anywhere after 16 whenever directed, bringing the framework reorg with it |
| 10 | 9b | [`@table`, relations, query](done/09b-table-relations-query.md) | `@table` real storage; `ref`/`backlink`/`multi`; compiler-checked queries |
| 11 | 15 | [deps: `wo.toml [deps]`](done/15-deps-package-manager.md) | exact-rev git deps + `wo.lock` + `.wo-deps`; flat-only, offline once locked |
| 12 | 16 | [web framework](done/16-web-framework.md) | the `.wo` framework v1 (router, middleware, auth, all three body hooks) consumed via `[deps]` |
| 13 | 8+11 | [Shard-actor runtime](08-shard-actor-runtime.md) · [Fibers](refine/11-fibers.md) | THE ARC (stages 1+2 landed 2026-08-20: fibers/budget/actors/io_uring plane; pinned shards, envelope sends, home-routed frees, WO-E222); stage 3 = transparent DB RPC + 22 re-run |
| 14 | 18 | [framework v2: memory-rich features](18-memory-db-features.md) | spec+plan approved: TTL cache, @table flags, durable job queue, `transaction { }` over the WAL's staged batch |
| 15 | 19 | [Float + Bytes](19-missing-scalar-types.md) | the missing scalars, full stack: IEEE-quiet f64 through literals/VM/@table/WAL/json + Bytes as the binary carrier — feeds 24 (WS frames) and the crypto fork (digests). *(was 20)* |
| 16 | 20 | [Cross-program tables](refine/20-cross-program-tables.md) | attach to a running program's database over local IPC; owner stays the single writer (channel half-built). *(was 9c)* |
| 17 | 21 | [Keypair attach auth](refine/21-keypair-attach-auth.md) | program identity is a keypair; mutual challenge–response at attach (crypto half-built; plan folds into 20's). *(was 9d)* |
| 18 | 22 | [Durability, throughput, scale](refine/22-durability-throughput-scale.md) | restart-persistence proof, benchmarks, ~1M rows — the baseline the arc and 23 sign against. *(was 9e)* |
| 19 | 23 | [io_uring group-commit](refine/23-io-uring-commit.md) | WAL WRITE+FSYNC chains on the arc's per-shard rings; fsync fallback kept (after 22 + the arc). *(was 9f)* |
| 20 | 24 | [chat: WebSocket workload](refine/24-chat-websocket-workload.md) | the arc's acceptance: WS upgrade + frames (SHA-1 via crypto fork, Bytes via 19), rooms/broadcast, 1k clients, drain-clean. *(was 19)* |
| 21 | 25 | [HTTP service layer](25-http-service.md) | `service` blocks lower onto the framework (after 9b + 20 by their own precedence notes). *(was 10)* |
| 22 | 26 | [Blue-green deploy](26-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback (plan authored after 9 + 25). *(was 12)* |
| 23 | 27 | [Query grammar corpus](refine/27-query-grammar-corpus.md) | grow the query grammar from real corpora; likely collapses to "confirm `len(query)` + add `exists`"; precedes 28. *(was 9g)* |
| 24 | 28 | [skillhost host workload](refine/28-skillhost-host-workload.md) | host-shaped driving workload naming runtime gaps — demoted with the framework goal. *(was 14)* |
| 25 | 29 | [Compile-time metaprogramming](refine/29-compile-time-metaprogramming.md) | `@derive(...)` from class-table metadata; held with the parked drain by the 2026-08-08 scope directive. *(was 13)* |
| ⏸ | 17 | [library projects + `internal/`](17-library-projects-internal.md) | **PARKED** (spec + plan approved, branch `library-internal`) — `wo.toml` kind = "library" + Go's `internal/` rule; slots anywhere after 16 on directive |
Review protocol: the developer reads one iteration, approves or amends;
the next starts only after approval. Each iteration is an unsplittable
@ -104,8 +110,8 @@ list, and a pointer to the plan document that already sequences its tasks.
**parked** with spec + plan ready on branch `library-internal`. The
v1 slices landed 2026-08-20 (`just web-app` 21/0 after polish, auth,
form, multipart). Implementation order for everything still pending:
**18 (spec approved)** → 9c/9d → 9e → 8 → 9f → 11 (+ h2c unparks) →
10 → 12 → 9g → 14 → 13 + parked drain; 17 parked, slots anywhere after
**18 (spec approved)** → 20/21 → 22 → 8 → 23 → 11 (+ h2c unparks) →
10 → 12 → 27 → 14 → 13 + parked drain; 17 parked, slots anywhere after
16 on directive. The iterations table above carries this order
row-by-row; edges live in
[`docs/00-dependency-graph.md`](../../00-dependency-graph.md).

4
docs/stories/language-runtime-database/08-shard-actor-runtime.md
View file

@ -45,9 +45,9 @@
- **Gated by the benchmark (2026-08-15):** this is the "optimize
multithreading" lever of the performance arc — thread-per-core is a
throughput/scale claim, so landing it means re-running iteration
[9e](refine/09e-durability-throughput-scale.md) at the connection/concurrency
[22](refine/22-durability-throughput-scale.md) at the connection/concurrency
scale it unlocks and recording the before/after delta. It is also where
the io_uring write path ([9f](refine/09f-io-uring-commit.md)) gets a thread to
the io_uring write path ([23](refine/23-io-uring-commit.md)) gets a thread to
overlap durability against.
## Proposed Solution

4
docs/stories/language-runtime-database/18-memory-db-features.md
View file

@ -53,8 +53,8 @@ worked around.
writes. Honest limit stated everywhere it matters: an idle server
drains nothing until the next request arrives. Fibers (11) later
replaces the scheduler; the queue table and job shape stay.
(Rejected for v1: a second worker process over 9c attach — real
parallelism but blocks on finishing 9c; parking jobs entirely — the
(Rejected for v1: a second worker process over 20 attach — real
parallelism but blocks on finishing 20; parking jobs entirely — the
queue-plus-drain is useful today.)
3. **`transaction { }` ships in this iteration.** Language block deferring
`wal_commit` to block end; a trap unwinding out of the block aborts the

6
docs/stories/language-runtime-database/20-missing-scalar-types.md → docs/stories/language-runtime-database/19-missing-scalar-types.md
View file

@ -1,4 +1,4 @@
# Iteration 20 — the missing scalar types: Float and Bytes
# Iteration 19 — the missing scalar types: Float and Bytes
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](00-story.md).
@ -30,9 +30,9 @@ work, but it blurs every json/interpolation boundary Text has.
and `trunc(f)` are the explicit bridges.
3. **Bytes ships alongside**: a distinct binary scalar (len/byte_at/
slice/compare; base64 and future digests return it; net reads can
fill it) — Text goes back to meaning text; iteration 19 and the
fill it) — Text goes back to meaning text; iteration 24 and the
crypto fork inherit a clean carrier.
4. **Recorded as iteration 20; spec before code.**
4. **Recorded as iteration 19; spec before code.**
## Surveyed and deliberately NOT added (the rest of the missing-type list)

2
docs/stories/language-runtime-database/10-http-service.md → docs/stories/language-runtime-database/25-http-service.md
View file

@ -1,4 +1,4 @@
# Iteration 10 — HTTP service layer
# Iteration 25 — HTTP service layer
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](00-story.md).

2
docs/stories/language-runtime-database/12-blue-green-deploy.md → docs/stories/language-runtime-database/26-blue-green-deploy.md
View file

@ -1,4 +1,4 @@
# Iteration 12 — blue-green in-runtime deployment
# Iteration 26 — blue-green in-runtime deployment
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](00-story.md).

2
docs/stories/language-runtime-database/done/07b-inferred-gc-mark-sweep.md
View file

@ -92,7 +92,7 @@
- **Gated by the benchmark (2026-08-15):** this is the "implement garbage
collection" lever of the performance arc — tri-color mark-sweep replacing
RC changes the write path's tail latency, so landing it means re-running
iteration [9e](../refine/09e-durability-throughput-scale.md) and recording the
iteration [22](../refine/22-durability-throughput-scale.md) and recording the
delta (does tracing help or hurt p99 under write load?).
- **Constraint added by the database track (2026-08-15):** a GC-managed value
in a `@table` field is a compile error (the engine/heap bulkhead — 9b

4
docs/stories/language-runtime-database/done/09b-table-relations-query.md
View file

@ -5,7 +5,7 @@
>
> **Inserted 2026-08-11**, hence `9b` rather than a renumber. It follows
> iteration 9 because a query surface needs tables that actually execute, and
> precedes iteration 10 because `service` blocks will want to return query
> precedes iteration 25 because `service` blocks will want to return query
> results.
>
> **Spec exists (2026-08-15):**
@ -74,7 +74,7 @@
HTTP/UI track's; a query that pushes updates is a later composition of the
two.
- Migrations. Changing a `@table` class's shape is the blue-green spec's
additive-only differ (iteration 12), not this iteration's problem.
additive-only differ (iteration 26), not this iteration's problem.
- Query optimisation beyond index selection. A cost-based planner is a
separate, much later concern; this iteration must only prove that declared
indexes are used.

8
docs/stories/language-runtime-database/done/16-web-framework.md
View file

@ -58,7 +58,7 @@
> like any dependency (iteration 15 is the prerequisite). TLS terminates at a
> reverse proxy — browsers get TLS+ALPN+h2 from nginx/caddy while the
> framework speaks HTTP/1.1 keep-alive behind it, so no TLS exists anywhere
> in the toolchain. h2c is the parked successor (after iterations 8/9f/11,
> in the toolchain. h2c is the parked successor (after iterations 8/23/11,
> when multiplexing has a scheduler to pay off on).
>
> **Spec exists:** [`2026-08-18-web-framework-design.md`](../../../superpowers/specs/2026-08-18-web-framework-design.md)
@ -88,7 +88,7 @@
(concurrency arrives underneath via iterations 8/11), no chunked encoding,
no WebSockets, JSON-first (no templates — the removed UI track stays
removed).
- **Relationship to iteration 10 recorded in both**: `service` blocks later
- **Relationship to iteration 25 recorded in both**: `service` blocks later
*lower onto this library* — compiler sugar over the same router, never a
rival stack.
@ -111,9 +111,9 @@
## Out Of Scope
TLS in the toolchain (proxy-terminated by decision); HTTP/2 + the
bytes/buffer type (parked to the h2c successor, after 8/9f/11); chunked
bytes/buffer type (parked to the h2c successor, after 8/23/11); chunked
transfer encoding; WebSockets/SSE; templates/SSR; multipart uploads;
performance work beyond the soak's flatness gate (benchmarks belong to 9e's
performance work beyond the soak's flatness gate (benchmarks belong to 22's
measurement backbone).
## Proposed Solution

2
docs/stories/language-runtime-database/refine/11-fibers.md
View file

@ -39,7 +39,7 @@
- **Given** a parked fiber at shard shutdown,
- **when** the shard unwinds it,
- **then** every drop map runs (ASan zero leaks) — parked fibers die
as cleanly as trapped ones. (Iteration 12's blue-green drain reuses
as cleanly as trapped ones. (Iteration 26's blue-green drain reuses
exactly this unwind path.)
- What to achieve?
- **Given** `@gc` objects referenced only from a parked fiber's frames,

14
docs/stories/language-runtime-database/refine/09c-cross-program-tables.md → docs/stories/language-runtime-database/refine/20-cross-program-tables.md
View file

@ -1,12 +1,12 @@
# Iteration 9c — cross-program tables: attach to a running program's database
# Iteration 20 — cross-program tables: attach to a running program's database
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).
>
> **Inserted 2026-08-15**, hence `9c`. It follows 9b because a program
> **Inserted 2026-08-15**, hence `20`. It follows 9b because a program
> attaching to another's tables wants the same typed statements and queries
> the owner has — a surface that must exist before it can be shared — and
> precedes iteration 10 because HTTP is the *external* face of a program;
> precedes iteration 25 because HTTP is the *external* face of a program;
> this iteration is the *writeonce-native* face, program to program on the
> same machine.
>
@ -72,7 +72,7 @@
## Out Of Scope
- **Remote machines.** The IPC string names a local channel; cross-host
access is the HTTP/service layer's job (iteration 10) or a much later
access is the HTTP/service layer's job (iteration 25) or a much later
network protocol. Same-machine is what "attach" means here.
- **B caching A's rows.** Every read crosses the channel; a client-side
cache (and its invalidation) is a later performance iteration, if ever.
@ -84,7 +84,7 @@
subscription registry later (the client-api phase doc already sketches
the wire shape).
- **Schema migration while attached** — a blue-green swap in A while B
holds an attachment is iteration 12's compatibility problem; this
holds an attachment is iteration 26's compatibility problem; this
iteration may simply drop attachments on swap.
## Info
@ -136,8 +136,8 @@ read or read+write (per-table refinement deferred until a workload needs
it), and the registration is A's manifest so a grant is a config change +
restart, not an API. **Superseded as the end state (2026-08-15):**
identity is a keypair and grants name public keys — iteration
[9d](09d-keypair-attach-auth.md) owns that; the uid check is only this
iteration's bootstrap and must be flagged pre-9d wherever it ships.
[21](21-keypair-attach-auth.md) owns that; the uid check is only this
iteration's bootstrap and must be flagged pre-21 wherever it ships.
**4. What does B's statement actually block on?** B's insert crosses the
channel, executes in A (RAM + WAL + fsync), and acknowledges back — a

22
docs/stories/language-runtime-database/refine/09d-keypair-attach-auth.md → docs/stories/language-runtime-database/refine/21-keypair-attach-auth.md
View file

@ -1,13 +1,13 @@
# Iteration 9d — keypair authentication for cross-program attach
# Iteration 21 — keypair authentication for cross-program attach
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).
>
> **Inserted 2026-08-15.** Promotes iteration 9c's identity fork (Info,
> **Inserted 2026-08-15.** Promotes iteration 20's identity fork (Info,
> fork 3) to its own iteration: the name + unix-uid lean is the milestone
> bootstrap, and THIS is what replaces it — program identity is a keypair,
> and an attachment is granted to a public key, not to a process that
> happens to share a uid. It follows 9c (there is nothing to authenticate
> happens to share a uid. It follows 20 (there is nothing to authenticate
> until attach exists) and stays same-machine; the same handshake is what
> a future remote channel would reuse, which is the point of doing it
> properly now.
@ -21,7 +21,7 @@
manifest) and a public key it can print/export. Identity stops being
"whoever reached the socket first with the right uid".
- **Grants name public keys.** A's `[share]` registers a client by its
public key (fingerprint), with rights exactly as 9c defined them; B's
public key (fingerprint), with rights exactly as 20 defined them; B's
`[connect.a]` **pins A's public key** beside the IPC string. Both sides
authenticate: A proves it is A before B sends a byte of intent, B proves
it is B before A executes a statement.
@ -38,7 +38,7 @@
read+write, and B's `[connect.a]` pinning A's public key,
- **when** B attaches,
- **then** the mutual handshake completes, the attachment carries B's
granted rights, and every 9c acceptance behavior (statements, traps,
granted rights, and every 20 acceptance behavior (statements, traps,
refusals) holds unchanged on top of it.
- What to achieve?
- **Given** a client presenting a keypair A never registered,
@ -47,7 +47,7 @@
client sees the catchable authentication trap, and A logs the offered
fingerprint (so granting it is a copy-paste, not an investigation).
- What to achieve?
- **Given** a same-uid process (the 9c bootstrap's whole trust basis)
- **Given** a same-uid process (the 20 bootstrap's whole trust basis)
presenting no key or the wrong key,
- **when** it attempts to attach,
- **then** it is refused — proving the uid check has been superseded,
@ -128,11 +128,11 @@ authorization input.
## Proposed Solution
- **Brainstorm the spec** settling the four forks, then fold the plan into
9c's implementation plan as its authentication tasks — one plan, because
9c without 9d ships a placeholder identity and 9d without 9c has nothing
to authenticate. The 9c milestone may still land first with the uid
bootstrap, flagged loudly as pre-9d.
- **Acceptance extends the 9c workload**: the employee-A /
20's implementation plan as its authentication tasks — one plan, because
20 without 21 ships a placeholder identity and 21 without 20 has nothing
to authenticate. The 20 milestone may still land first with the uid
bootstrap, flagged loudly as pre-21.
- **Acceptance extends the 20 workload**: the employee-A /
employee-list-B pair (`docs/examples/employee-list`, pre-authored
2026-08-15) carries the key exchange in both manifests — A's
`[[share.clients]]` names B's fingerprint, B's `[connect.employee]` pins

22
docs/stories/language-runtime-database/refine/09e-durability-throughput-scale.md → docs/stories/language-runtime-database/refine/22-durability-throughput-scale.md
View file

@ -1,4 +1,4 @@
# Iteration 9e — durability proof, throughput, and scale under load
# Iteration 22 — durability proof, throughput, and scale under load
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).
@ -6,7 +6,7 @@
> **Inserted 2026-08-15.** The measurement backbone. Everything after the
> functional engine (9/9b) is an *optimization*, and an optimization without
> a number is a guess — this iteration is the number. It comes before the
> optimization iterations (7b GC, 8 shard-actor, 9f io_uring) reopen for
> optimization iterations (7b GC, 8 shard-actor, 23 io_uring) reopen for
> performance work, because each of those must be gated by re-running THIS
> iteration's benchmark and showing the number moved the right way.
>
@ -29,7 +29,7 @@
a stated duration, with throughput and tail latency inside a stated budget
and RSS flat (the log-watcher soak discipline, at database scale).
- **The benchmark is the contract every later optimization signs.** 7b (GC),
8 (shard-actor threads), and 9f (io_uring) each re-run this and record the
8 (shard-actor threads), and 23 (io_uring) each re-run this and record the
before/after — no optimization lands without a measured delta.
## Acceptance Criteria
@ -58,14 +58,14 @@
a durable configuration — a kill mid-load followed by replay loses no
acknowledged write.
- What to achieve?
- **Given** any later optimization iteration (7b, 8, 9f),
- **Given** any later optimization iteration (7b, 8, 23),
- **when** it claims a speedup,
- **then** this benchmark's before/after numbers are in that iteration's
record, and a claim with no measured delta is not accepted.
## Out Of Scope
- **The optimizations themselves.** This iteration MEASURES; 7b/8/9f change.
- **The optimizations themselves.** This iteration MEASURES; 7b/8/23 change.
A single-thread RAM-authoritative baseline is a legitimate first number —
the point is to have one before anyone tunes.
- **Distributed / multi-machine load.** Same-machine, one process (or one
@ -114,7 +114,7 @@ a bounded concurrent read/write workload here; connection scale deferred to
the honest number for a durable workload; RAM-only (no `WO_DATA`) measures
the engine's ceiling. Both matter and mean different things. Leaning:
publish both, labeled — durable is the number an operator plans against, and
the gap between them is precisely what iteration 9f (io_uring group-commit)
the gap between them is precisely what iteration 23 (io_uring group-commit)
exists to close.
## Proposed Solution
@ -123,13 +123,13 @@ exists to close.
task is the harness and the baseline file, because nothing downstream means
anything without them.
- **Sequence the whole performance arc around this iteration:**
1. 9b lands → employee compiles and runs → **9e restart-persistence** and
**9e baseline benchmark** (single-thread, both durable and RAM-only).
2. **7b** (inferred GC + mark-sweep) → re-run 9e, record the delta (does
1. 9b lands → employee compiles and runs → **22 restart-persistence** and
**22 baseline benchmark** (single-thread, both durable and RAM-only).
2. **7b** (inferred GC + mark-sweep) → re-run 22, record the delta (does
tracing change the write path's tail latency?).
3. **8** (shard-actor, thread-per-core) → re-run 9e at the connection/
3. **8** (shard-actor, thread-per-core) → re-run 22 at the connection/
concurrency scale it unlocks, record the delta.
4. **9f** (io_uring group-commit) → re-run 9e's durable write number, record
4. **23** (io_uring group-commit) → re-run 22's durable write number, record
the delta against the fsync-per-commit baseline — the payoff.
- The benchmark harness and its baseline live under `bench/` (or the existing
`runtime/bench/`), and `just` gets a `db-bench` recipe kept off the fast

22
docs/stories/language-runtime-database/refine/09f-io-uring-commit.md → docs/stories/language-runtime-database/refine/23-io-uring-commit.md
View file

@ -1,11 +1,11 @@
# Iteration 9f — io_uring group-commit write path
# Iteration 23 — io_uring group-commit write path
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).
>
> **Inserted 2026-08-15.** The write-path optimization, and deliberately the
> LAST database performance iteration: it only earns its complexity once
> there is a measured fsync-per-commit baseline to beat (iteration 9e) and a
> there is a measured fsync-per-commit baseline to beat (iteration 22) and a
> multithreaded runtime to overlap against (iteration 8). Doing it earlier
> would optimize a number nobody had measured, against a runtime that
> couldn't use it.
@ -24,21 +24,21 @@
statements while the ring drains, instead of blocking one thread on one
fdatasync — the multithreading the throughput number has been waiting for.
- **Keep the durability promise byte-for-byte.** Every guarantee iterations 9
and 9e proved — replay-whole-or-not-at-all, torn-tail drop, no
and 22 proved — replay-whole-or-not-at-all, torn-tail drop, no
acknowledged write ever lost — holds identically; io_uring changes HOW the
bytes reach the platter, never WHETHER an ack means durable.
## Acceptance Criteria
- What to achieve?
- **Given** the io_uring write path under the iteration-9e crash battery
- **Given** the io_uring write path under the iteration-22 crash battery
(concurrent writers, kill -9 mid-stream, reboot, replay),
- **when** it runs,
- **then** every acknowledged write is present after replay and no
unacknowledged partial write is ever visible — the exact result the
fsync path gives, so durability is provably unchanged.
- What to achieve?
- **Given** the iteration-9e durable write benchmark,
- **Given** the iteration-22 durable write benchmark,
- **when** it is run on the fsync-per-commit path and then the io_uring
group-commit path on the same machine,
- **then** the io_uring path's write throughput is materially higher and
@ -61,7 +61,7 @@
read path. This is a write-durability optimization, full stop.
- **Registered buffers / fixed files / SQPOLL tuning** beyond what the
benchmark shows is worth it. Start with the plain submit/complete model;
add ring features only when 9e's number says a specific one pays.
add ring features only when 22's number says a specific one pays.
- **Replacing the WAL format or the commit contract.** The bytes on disk and
the meaning of an ack are iteration 9's; this changes the syscall, not the
format.
@ -78,7 +78,7 @@ wrap it behind the existing `wo_wal_commit` boundary (drop-in, the engine
never learns) or expose an async-commit primitive the shard scheduler drives
(faster overlap, but couples the WAL to iteration 8's loop). Leaning:
drop-in behind `wo_wal_commit` first — it is the correctness-preserving
step and 9e can measure it standalone — then an async variant only if 8's
step and 22 can measure it standalone — then an async variant only if 8's
scheduler shows the blocking boundary is the remaining bottleneck.
**2. liburing or raw syscalls?** liburing is the ergonomic wrapper but is a
@ -104,13 +104,13 @@ auto-probe is what production uses.
## Proposed Solution
- **Brainstorm the spec** after iterations 8 and 9e exist — this iteration is
- **Brainstorm the spec** after iterations 8 and 22 exist — this iteration is
meaningless without a multithreaded runtime to overlap against and a
measured baseline to beat, and its plan's acceptance is literally "9e's
durable number improved, 9e's crash battery still green, fsync fallback
measured baseline to beat, and its plan's acceptance is literally "22's
durable number improved, 22's crash battery still green, fsync fallback
still correct".
- Expected shape: a `wo_wal` write-mode switch (fsync vs uring), the raw ring
setup + submit/complete in `database/src/wal.c` (or a `wal_uring.c`
beside it), the startup probe + `WO_WAL_MODE` override, the binding doc's
WAL section extended with the ring layout, and iteration 9e re-run on both
WAL section extended with the ring layout, and iteration 22 re-run on both
paths with the delta committed.

69
docs/stories/language-runtime-database/refine/24-chat-websocket-workload.md Normal file
View file

@ -0,0 +1,69 @@
# Iteration 24 — chat: the WebSocket pub/sub driving workload
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).
>
> **Inserted 2026-08-20** (concurrency-chain refinement): the 8+11 arc's
> driving workload, the role log-watcher played for iterations 3–7. Needs
> its spec AFTER the arc's — it lands at the arc's end and proves it.
## Why this iteration exists
Everything the framework ledger parks behind concurrency — WebSockets,
pub/sub, streaming, per-request cancellation, the keep-alive parking
retirement — needs a workload that actually exercises long-lived
connections and cross-shard broadcast, or the arc ships mechanism without
proof. Chat is the smallest honest such workload: rooms, N concurrent
clients, fan-out on every message, presence on connect/disconnect — one
binary, no broker.
## Goals
- `docs/examples/chat`: rooms + broadcast + presence over WebSocket,
served by the framework through `[deps]` exactly as the web-app is.
- Framework grows the WS mechanism in pure `.wo`: the HTTP/1.1 upgrade
handshake (Sec-WebSocket-Accept needs SHA-1/base64 — the crypto-builtin
fork's first real consumer), frame parse/serialize (text, close, ping),
fiber-per-connection serving (iteration 11), and in-memory channels
whose delivery is cross-shard message send (iteration 8).
- The arc's acceptance teeth: 1k concurrent clients across shards,
broadcast latency measured, starvation-free under one hot room,
SIGTERM drains every connection cleanly.
## Acceptance Criteria (draft — the spec after the arc refines)
- **Given** two clients in one room on DIFFERENT shards, **when** one
sends, **then** the other receives the frame (cross-shard ownership-move
delivery), and a third client in another room receives nothing.
- **Given** 1k connected clients with one hot sender, **when** the
reduction budget preempts, **then** every room keeps making progress
(no starvation) on one OS thread per core, verified by TID.
- **Given** SIGTERM with clients connected, **when** the server drains,
**then** every connection gets a close frame, every fiber unwinds its
drop maps (ASan zero leaks), and the process exits 0.
- **Given** the web-app running beside chat features, **when** the
standing gates run, **then** nothing regresses — HTTP and WS share the
serve loop honestly.
## Out Of Scope
Message persistence/history (a `@table` an app adds if it wants — not the
workload's point); auth beyond the existing bearer mechanism; permessage
compression; binary frames beyond echo coverage; wss (TLS stays at the
proxy — the proxy story extends to WS pass-through, documented).
## Info
- Dependencies: the 8+11 arc (fibers + cross-shard send), the crypto
builtins fork (SHA-1 for the upgrade handshake — note: the ledger's
crypto slice lists SHA-256/512; the WS handshake specifically needs
SHA-1, so the builtin set must include it), and the framework's parse
seam (upgrade is an HTTP request until it isn't).
- Unparks on landing: the framework ledger's WebSocket/pub-sub rows and
the iteration-18 rejection note ("pub/sub REJECTED until 8/11").
## Proposed Solution
Brainstorm → spec → plan after the arc's spec exists; the arc's plan and
this iteration's are written against each other (the arc names chat as
its acceptance, chat names the arc as its substrate).

2
docs/stories/language-runtime-database/refine/09g-query-grammar-corpus.md → docs/stories/language-runtime-database/refine/27-query-grammar-corpus.md
View file

@ -1,4 +1,4 @@
# Iteration 9g — query grammar, driven by real embedded-DB corpora
# Iteration 27 — query grammar, driven by real embedded-DB corpora
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).

8
docs/stories/language-runtime-database/refine/14-skillhost-host-workload.md → docs/stories/language-runtime-database/refine/28-skillhost-host-workload.md
View file

@ -1,4 +1,4 @@
# Iteration 14 — skillhost: a host-shaped workload, and the capability gaps it exposes
# Iteration 28 — skillhost: a host-shaped workload, and the capability gaps it exposes
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).
@ -34,7 +34,7 @@
## What is already expressible (verified 2026-08-16)
- **The skill catalog** — `@table` with the query surface already exceeds
skillhost's in-memory SQLite `skills` table; iteration 9g's
skillhost's in-memory SQLite `skills` table; iteration 27's
`docs/examples/skill-catalog` is literally this table, running. (Or a plain
`map`/`multi` would do — the catalog is a lookup cache, not persistence.)
- **Discovery** — `fs.exists`/`fs.list` (one level) + `fs.read_all` walk
@ -65,7 +65,7 @@ Two directions, and they are a real fork, not a detail:
- **FFI as a language capability** — a way to declare and call C functions
from writeonce. This is a large, doctrine-level addition (the runtime is
libc-only by principle; the one sanctioned exception so far is the vendored
Ed25519, 9d). FFI would reopen the dependency-sprawl question the whole
Ed25519, 21). FFI would reopen the dependency-sprawl question the whole
project is built to avoid. Likely its own spec, likely contested.
- **Out-of-process model, no FFI** — drive a llama.cpp binary via `proc`
(`llama-cli`) or `llama-server` over `net` + `json` (it accepts a GBNF
@ -194,6 +194,6 @@ then B and the partials narrow the gap to skillhost's real behavior.
exactly the open gaps (A: out-of-process model, B: socket not stdio, C:
bounded once its iteration lands, plus the fs partials) — the list is the
iteration's own scoreboard.
- Reuse iteration 9g's `skill-catalog` as the catalog layer, log-watcher's
- Reuse iteration 27's `skill-catalog` as the catalog layer, log-watcher's
MCP mode as the transport skeleton, and the systems stdlib for discovery
and execution.

4
docs/stories/language-runtime-database/refine/13-compile-time-metaprogramming.md → docs/stories/language-runtime-database/refine/29-compile-time-metaprogramming.md
View file

@ -1,4 +1,4 @@
# Iteration 13 — compile-time metaprogramming (derive from the class table)
# Iteration 29 — compile-time metaprogramming (derive from the class table)
> Format: `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).
@ -99,7 +99,7 @@ rather than against it.
- **Monomorphized generics as a general feature.** Per-type generation here is
specific to the derive set, not a general generics engine.
- **Deriving across the attach channel** — a client generating an encoder over
the owner's types (iterations 9c/9d). Composes later; the class-table
the owner's types (iterations 20/21). Composes later; the class-table
metadata already crosses the channel's schema handshake, so the pieces are
in place, but it is not this iteration's problem.
- **Reopening principle 13 in any form.** If a derive appears to need runtime

2
docs/superpowers/plans/2026-08-01-http-service-layer.md
View file

@ -1,6 +1,6 @@
# HTTP Service Layer Implementation Plan
> **Status: ⬜ pending** (story iteration 10) — `service` blocks route to VM methods; REST parity with the shipped Rust Stage 2 runtime. Board: [00-status.md](../../00-status.md)
> **Status: ⬜ pending** (story iteration 25) — `service` blocks route to VM methods; REST parity with the shipped Rust Stage 2 runtime. Board: [00-status.md](../../00-status.md)
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
>

349
docs/superpowers/plans/2026-08-20-framework-v2-memory-features.md Normal file
View file

@ -0,0 +1,349 @@
# Iteration 18 — framework v2 (transaction{} + cache/flags/jobs): implementation plan
> **Status: ready to execute (2026-08-20).** Board:
> [docs/00-status.md](../../00-status.md).
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or
> superpowers:executing-plans to implement this plan task-by-task. Steps
> use checkbox (`- [ ]`) syntax for tracking.
>
> **Style rule (user convention):** concept, reason, and required behavior
> in words plus verification commands only — no implementation or test
> code blocks; the executor writes the code.
**Goal:** `transaction { }` makes multi-statement writes one WAL commit
(trap = abort), and three framework pieces ride the single process:
a TTL cache, `@table` feature flags with a cached read, and a durable
`@table` job queue drained in-process — the web-app proves the headline
(order + job enqueue, ONE commit, SIGKILL-survivable).
**Architecture:** the engine already stages WAL batches
(`wal_append_*` → `wo_wal_commit`, per-statement today in
`database/src/db.c`'s three write cases); a transaction defers the commit
and keeps an undo log for RAM. The VM learns two internal builtins and a
transaction-flagged catch frame; the parser one keyword. Cache/flags/jobs
are pure `.wo` in the framework plus one serve-loop seam. No new opcodes,
no `.wob` version bump, GC untouched.
**Tech Stack:** C11 libc-only (engine + VM), OCaml stdlib-only (`woc`),
pure `.wo` (framework), bash gates.
**Spec:** [`../specs/2026-08-20-memory-db-features-design.md`](../specs/2026-08-20-memory-db-features-design.md)
(approved 2026-08-20, normative). Story:
[`18-memory-db-features.md`](../../stories/language-runtime-database/hold/18-memory-db-features.md).
## Global Constraints
- Branch `framework-v2` off master; commits local only, never push.
- Exit/trap doctrine unchanged: WO-E110 nested `transaction` (compile),
WO_T_DB "nested transaction" (dynamic, through a call); commit failure
is the existing WO_T_IO shape with RAM never ahead of disk.
- Gates that must stay green after every task: `just woc-test`,
`just oop-e2e`, `just deps-accept`, `just web-app`, `just log-watcher`,
`just employee` — plus the oop-accept ASan clause for anything the VM
touches.
- Framework tables carry the `wf_` prefix; policy stays app-side.
- `time.now` is wall-clock MILLISECONDS everywhere in this iteration.
## Spec deviations, disclosed up front
1. **`return`/`break`/`continue` crossing a `transaction { }` boundary is
rejected at compile time (new WO-E112).** The spec is silent on early
exit; commit-on-return vs abort-on-return is exactly the ambiguity a
v1 must not guess at. A later iteration may define it; today the block
has one entrance and one exit (a TRAP still aborts — that path is
defined).
2. **The cache exposes a time-injected seam** (`get_at`/`put_at` taking a
now-milliseconds argument, with `get`/`put` sugar reading `time.now`)
so the corpus fixture injects stamps instead of sleeping — the spec's
"stamps injected, not waited" made concrete.
3. **Flags are gate-proven, not corpus-proven**: a `@table` needs
`WO_DATA` and a persistent directory, which the corpus harness does
not provide; `just web-app` carries the flags checks (the spec's gate
section already put them there).
4. **The WAL gains an explicit staged-batch discard** (`wo_wal_abort`):
today a failed commit "stays staged" by contract; abort needs to drop
the batch deliberately. Same file, same batch machinery, new entry
point.
---
## Task 1 — engine transactions: defer commit, undo log, abort
**Files:**
- Modify: `database/src/db.h` (txn state on `wo_db`, three new entry
points), `database/src/db.c` (the three write cases ~lines 17–80),
`database/src/wal.h` + `wal.c` (`wo_wal_abort`),
`database/src/table.h` + `table.c` only if the replay-only fixed-id
create/index pair needs a non-static wrapper.
- Test: `runtime/test/test_txn.c` (new, mirroring the existing
`test_*.c` harness shape), wired into the runtime test recipe.
**Interfaces:**
- Produces: `wo_db_txn_begin(db)` (0 ok; nonzero = already open — the
dynamic-nesting signal), `wo_db_txn_commit(db, wal)` (the ONE
`wo_wal_commit`; on failure the undo walk runs and the error returns),
`wo_db_txn_abort(db, wal)` (reverse undo walk + `wo_wal_abort`).
Task 3's builtins call exactly these three.
- [ ] `wo_db` grows a transaction flag and an undo log (a growable array
of entries: op kind, class id, row id, and for update/delete a copy of
the row's bytes — `wo_row_ptr` + the class's `row_size` are the copy;
the log exists only while a transaction is open, zero cost otherwise).
- [ ] The three write cases in `wo_builtin_db`: when the flag is set,
record the undo entry BEFORE the RAM apply (update/delete pre-images;
insert records just the new id AFTER apply), still `wal_append_*`, and
SKIP the per-statement `wo_wal_commit`. Flag clear = byte-identical
behavior to today (every existing gate is the proof).
- [ ] Statement-level failures inside a transaction change nothing: a
unique violation traps before apply and stages nothing (already true —
`wo_row_insert` refuses first); a failed append keeps the pre-existing
error shape.
- [ ] Abort walks the undo log in REVERSE: inserted row → removed;
updated row → bytes restored and index entries fixed the replay way
(remove + re-add through the same engine-internal pair `wal.c` replay
uses); deleted row → re-created with its FIXED id and re-indexed (the
replay-only create), then `wo_wal_abort` discards the staged batch.
- [ ] Reads inside a transaction need no change: RAM stays applied, so
scans/point-reads see the block's own writes for free.
- [ ] `test_txn.c`: begin→insert+insert→commit = both rows + ONE wal
flush; begin→insert→abort = zero rows, next insert works; update and
delete pre-images restored on abort (indexed column included);
begin-while-open refused; abort with an empty log is a no-op.
- [ ] Run the runtime test suite + `just employee` (engine untouched when
no txn opens). Commit.
## Task 2 — the language surface: keyword, block, WO-E110/E112
**Files:**
- Modify: `compiler/src/token.ml` + `lexer.ml` (KwTransaction),
`compiler/src/ast.ml` (a Transaction statement holding a body),
`compiler/src/parser.ml` (block statement + LEXICAL nesting =
WO-E110), `compiler/src/types.ml` (walk the body; E-code constants),
`compiler/src/owner.ml` (treat as a plain nested scope),
`compiler/src/dump.ml` (labels).
**Interfaces:**
- Consumes: nothing new.
- Produces: the AST node Task 3 lowers; WO-E110 (parsing prefix, nested
block), WO-E112 (parsing prefix, `return`/`break`/`continue` whose
jump would cross the block boundary — a loop wholly INSIDE the block
keeps its own break/continue).
- [ ] Keyword + statement parse; the body is an ordinary statement list.
A `transaction` token while one is already open (parser-tracked depth)
is WO-E110 at the inner keyword.
- [ ] WO-E112: while parsing the block, a `return` at any depth, or a
`break`/`continue` not enclosed by a loop that itself started inside
the block, names the rule ("a transaction has one exit; lift the
return out or end the block first").
- [ ] Types/owner: the body checks exactly like a bare block — no new
typing rule (the ownership pass sees a scope; values born inside drop
inside, exactly as today).
- [ ] Compile-fail fixtures: `transaction-nested` (WO-E110),
`transaction-early-return` (WO-E112). Verify both + `just woc-test`
(dump labels) + `just oop-e2e`. Commit.
## Task 3 — VM lowering: internal builtins + the abort-on-unwind frame
**Files:**
- Modify: `runtime/src/wob.h` (two builtin ids in the internal range),
`runtime/src/builtin.c` (dispatch to Task 1's three entry points),
`runtime/src/vm.c` (transaction-flagged catch frame; the vm_trap walk;
unwind/rt-destroy cleanup), `compiler/src/emit.ml` (lower the
Transaction statement), `runtime/test/test_unwind.c` (frame cleanup on
a trap that leaves the whole method).
**Interfaces:**
- Consumes: Task 1's `wo_db_txn_begin/commit/abort`, Task 2's AST node.
- Produces: the observable spec semantics — one fdatasync at the closing
brace; trap unwinding OUT aborts and keeps unwinding; `try` INSIDE the
block keeps it alive.
- [ ] Two builtin ids the emitter emits directly from the Transaction
case (no name in any user-callable table — nothing to collide with):
begin pushes a TRANSACTION-FLAGGED catch frame (the existing TRY frame
machinery with one flag bit) and calls txn_begin (nonzero = the
WO_T_DB "nested transaction" trap); end calls txn_commit FIRST and
pops the frame only on success — a commit failure traps with the frame
still in place, so the abort path below runs and RAM is rolled back
(the "RAM never ahead of disk" rule, transaction-sized).
- [ ] vm_trap's handler search: a transaction-flagged frame is not a
handler — abort the transaction, pop it, CONTINUE searching. An inner
`try` frame sits ABOVE it and catches first (the spec's
inner-try-keeps-it-alive rule falls out of frame order, no special
case).
- [ ] Program exit / rt teardown with a transaction somehow open (a trap
that reaches main uncaught) must abort, not leak the undo log.
- [ ] Emitter: begin, body statements, end — plus the WO-E112 guarantee
from Task 2 meaning no jump ever leaves the region except a trap.
- [ ] Corpus: `run/transaction-commit` (two inserts, both rows readable
after — needs the trap corpus's WO_DATA-less shape? No: @table without
WO_DATA runs RAM-only with no WAL, which still exercises begin/commit
frames; the DURABILITY half lives in Task 6's gate where WO_DATA
exists), `run/transaction-abort` (second insert unique-traps, caught
OUTSIDE the block: first row gone too, inserts after the abort work,
process exits clean under ASan).
- [ ] `just oop-e2e` (ASan stage covers the new frames) + full battery.
Commit.
## Task 4 — framework cache: TTL + capacity, pure `.wo`
**Files:**
- Create: `docs/examples/writeonce-framework/store/cache.wo`.
- Test: `tests/corpus/run/cache-ttl/` (fixture copies the class inline —
corpus fixtures cannot `use` the framework; the framework file is the
same code verified by the framework's standalone compile + Task 7's
consumer build).
**Interfaces:**
- Produces: `pub class Cache { ttl_ms: Int, cap: Int, keys: multi Text,
vals: map<Text, Text>, stamps: map<Text, Int> }` with `get_at(now,
key) -> ?Text`, `put_at(now, key, value)`, and `get`/`put` sugar over
`time.now` — the shape apps hold as a field on any long-lived
instance.
- [ ] `get_at`: absent → nil; older than `ttl_ms` → remove the entry
(lazy expiry — there are no timers by design) and answer nil; live →
the value (caller-owned copy).
- [ ] `put_at`: store + stamp; when the key list exceeds `cap`, evict
OLDEST-INSERTED until within capacity (FIFO — the file states the
LRU tradeoff the spec settled). Re-putting an existing key refreshes
value + stamp without duplicating the key entry.
- [ ] Values are Text; the file says "json.encode structure into it" —
no generics exist, stated, not apologized for.
- [ ] Fixture drives injected stamps: fresh hit, expiry at exactly
ttl+1, eviction order under cap pressure, re-put refresh; ASan run.
- [ ] Framework standalone compile stays clean. Commit.
## Task 5 — framework flags: wf_flags + cached read-through
**Files:**
- Create: `docs/examples/writeonce-framework/store/flags.wo`.
**Interfaces:**
- Produces: `@table(name: "wf_flags")` class `Flag { name: Text @unique,
on: Int }` (Int 0/1 — Bool columns are unproven storage, spec's call)
and `pub class Flags { loaded: Int, cache: map<Text, Int> }` with
`read(name, default: Bool) -> Bool` and `set(name, on: Bool)`.
- [ ] `read`: first call fills the map from the table (query by name —
the employee-proven point-read), later calls answer from the map;
absent flag → the default, uncached (so a later `set` is seen).
- [ ] `set`: update-or-insert the row, then update the map in the same
call — single process, invalidation is an assignment. Durability is
the table's (WAL), restart rebuilds via `read`.
- [ ] Framework standalone compile; behavior proven in Task 7's gate
(deviation 3). Commit.
## Task 6 — framework jobs: wf_jobs, enqueue, JobRunner, the idle seam
**Files:**
- Create: `docs/examples/writeonce-framework/store/jobs.wo`.
- Modify: `docs/examples/writeonce-framework/http/serve.wo` (Dispatcher
gains `fn idle()`; the serve loop calls it after `net.accept`, BEFORE
parsing the connection's first request), `app.wo` (`App` satisfies
`idle`; `jobs(take r: Jr, budget: Int)` registration; `Jr { r:
JobRunner }` wrapper, the Mw/Route pattern), `README.md` (the drain
contract + the idle-server-drains-nothing disclosure).
**Interfaces:**
- Consumes: `transaction { }` (Tasks 1–3) only in the DEMO — enqueue
itself is an ordinary insert, composition happens in app code.
- Produces: `@table(name: "wf_jobs")` class `Job { kind: Text, payload:
Text, attempts: Int, not_before: Int }`; `pub fn enqueue(kind,
payload)`; `pub interface JobRunner { fn run(kind: Text, payload:
Text) -> Bool }`; `App.jobs(take r, budget)`; `Dispatcher.idle()`.
- [ ] Drain (in `App.idle`): no runner registered → return immediately.
Else query up to `budget` due jobs (`not_before <= time.now`,
registration order via `take`), each inside `try`: true → `delete`
the row; false or trap → `attempts + 1` (update), row stays —
retry/backoff policy is the app's (it can rewrite `not_before` from
its own runner).
- [ ] The post-accept/pre-parse placement is the DETERMINISM the gate
needs: a job enqueued by connection A never runs before A closes, and
a kill after A's response provably leaves the row. Latency cost
(≤ budget jobs ahead of the next request) stated in the README.
- [ ] Framework standalone compile; the serve loop's existing gates
(`just web-app` current count) stay green with NO runner registered —
the seam must cost nothing. Commit.
## Task 7 — the web-app demo + the gate
**Files:**
- Modify: `docs/examples/web-app/main.wo` (transactional CreateOrder +
confirm runner + `GET /jobs` count + `POST /flags/:name` + the
flag-gated header on the product list), `types.wo` (nothing — wf_
tables come from the framework), `README.md`,
`scripts/web-app-accept.sh`.
**Interfaces:**
- Consumes: everything above, through `[deps]` exactly like every other
framework feature.
- [ ] `CreateOrder.handle`: `transaction { insert Order {...};
enqueue("confirm", <order json>); }` — the headline composition, one
commit. A `Confirm` runner class answers `confirm` by printing the
order-confirmation line to stderr and returning true; registered via
`app.jobs(Jr { r: Confirm {...} }, budget)`.
- [ ] `GET /jobs` (behind the existing bearer auth): pending count as
JSON. `POST /flags/:name`: flips through `Flags.set`; the product
list answers an extra header (e.g. `x-store-banner`) while the flag
is on.
- [ ] Gate additions, in order: (a) `POST /orders` 201, then `kill -9`
the server IMMEDIATELY (no further requests), restart on the same
`WO_DATA`, then `GET /jobs` — the confirmation line appears in the
restarted server's log (the drain ran post-accept on this very
request) and the count answers 0: the job survived the kill because
it committed WITH the order; (b) `POST /flags/banner` then
`GET /products` carries the header, restart, still carries it;
(c) the standing matrix unchanged. Counts stay dynamic in the script.
- [ ] Full battery: `just web-app`, `woc-test`, `oop-e2e`,
`deps-accept`, `log-watcher`, `employee`. Commit.
## Task 8 — docs closeout
**Files:**
- Modify: `docs/00-status.md` (row 18 ✅ with measured results; NEXT
PLAN advances to 20/21 per the order), story `18-memory-db-features.md`
(landing banner) then `git mv` into `stories/.../done/` with links
re-pathed and VERIFIED, `docs/00-dependency-graph.md` (node classes:
18 done; TPRMW unblocks), framework `README.md` (ledger rows:
storage-integration txn-per-request now buildable; the v2 pieces ✅),
`compiler/src/CODE-LOGIC.md` + `runtime/src/CODE-LOGIC.md` +
`database/src/CODE-LOGIC.md` (the txn seams, one paragraph each).
- [ ] Apply; run `just web-app` once more after doc edits; commit.
## Success criteria (spec, restated as the gate reads them)
1. Two inserts in one block: SIGKILL before the next request → both rows
after restart (gate a); a trap unwinding out → neither row and the
process keeps serving (`run/transaction-abort` + ASan).
2. The order's job runs after the NEXT accepted connection within
budget, never before the posting connection closes, and survives a
kill in between (gate a).
3. An expired or evicted cache entry answers nil with no timer having
existed (`run/cache-ttl`, stamps injected).
4. A flipped flag holds across restart (gate b). Every standing gate
green; opcode set and `.wob` format byte-identical.
## Self-review notes
- Spec coverage: Part A semantics → Tasks 1–3 (observable rules mapped
one-to-one; the early-exit hole closed by deviation 1); cache → T4;
flags → T5; jobs + seam → T6; demo + gate → T7; out-of-scope list
untouched. Corpus/gate split follows deviations 2–3.
- Type consistency: the three engine entry points, the two E-codes
(E110/E112), `wf_flags`/`wf_jobs`, `get_at`/`put_at`,
`JobRunner.run(kind, payload) -> Bool`, `App.jobs(take r, budget)`,
`Dispatcher.idle()` — spelled identically in every task that names
them.
- Risk, disclosed: the abort walk's index restoration is the one place
correctness is subtle (indexed column updated then aborted); Task 1's
unit test pins exactly that case before any VM work stacks on it.
- Ordering: engine (T1) before VM (T3) with the language (T2) between so
T3 has both; cache/flags (T4/T5) are independent and could land any
time, kept after the critical path so the risky work gets the freshest
attention.

267
docs/superpowers/plans/2026-08-20-library-kind-internal.md Normal file
View file

@ -0,0 +1,267 @@
# Iteration 17 — library kind + `internal/`: implementation plan
> **Status: ⏸ PARKED 2026-08-20** (developer directive: framework v1 work
> proceeds instead; this plan stays ready on branch `library-internal`,
> execution not started). Board: [docs/00-status.md](../../00-status.md).
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or
> superpowers:executing-plans to implement this plan task-by-task. Steps use
> checkbox (`- [ ]`) syntax for tracking.
>
> **Style rule (user convention):** this plan carries concept, reason, and
> required behavior in words plus verification commands only — no
> implementation or test code blocks; the executor writes the code.
**Goal:** `kind = "library"` in `wo.toml` makes a project checkable without
an entry, the `internal/` rule keeps a dependency's plumbing private
(WO-E108), and the framework adopts both — `just web-app` proves the public
surface unmoved.
**Architecture:** every change is compile-time and lives in the driver
(`compiler/bin/main.ml`): the manifest reader learns one key, the manifest
build path grows a check branch, and the dep-use resolution walk in
`compile_image` grows the boundary rule. No lexer, parser, typechecker, VM,
`.wob`, or GC change — the spec's impact analysis is normative.
**Tech Stack:** OCaml stdlib only (`woc`), bash acceptance scripts, `just`.
**Spec:** [`../specs/2026-08-20-library-kind-internal-design.md`](../specs/2026-08-20-library-kind-internal-design.md)
(normative). Story:
[`17-library-projects-internal.md`](../../stories/language-runtime-database/17-library-projects-internal.md).
## Global Constraints
- Branch `library-internal`; commits local only, never push.
- OCaml stdlib only; no new executables, no new build steps.
- Absent `kind` means program — every existing project's behavior is
byte-identical; the standing gates prove it each task.
- Exit-code doctrine stays: 0 clean, 1 diagnostics, 2 usage/manifest/IO.
WO-E108 is a diagnostic (exit 1, printed by the normal collector path);
WO-E109 is a manifest error (exit 2, `woc: <file>: error WO-E109: ...`,
the WO-E106 shape).
- `internal` matches a whole path SEGMENT of a dep-relative module path —
never a substring.
- Gates that must stay green after every task: `just woc-test`,
`just oop-e2e`, `just deps-accept`, `just web-app`, `just log-watcher`,
`just employee`.
## Spec deviations, disclosed up front
1. The spec put WO-E108/E109 fixtures in the compile-fail corpus. The
corpus harness (`scripts/oop-e2e.sh`) compiles a fixture DIRECTORY with
no manifest and no network — a `wo.toml` in a fixture dir would flip woc
into manifest-build mode, and WO-E108 additionally needs a fetched dep.
Both diagnostics are therefore gated in `scripts/web-app-accept.sh`
(which already builds a file:// dep chain), not the corpus. Task 5.
2. The spec says check mode "stops before image emission". The cheapest
correct implementation reuses `compile_image` whole — emission happens
in memory and the image is discarded; no artifact is written, no entry
is required (an entry-less image is already legal there, the `--emit`
precedent). Behavior matches the spec; the pipeline boundary is one
step later than the spec's wording.
3. `woc build <dir> -o <out>` never reads the manifest, so it resolves no
`[deps]` (pre-existing, iteration 15). The dual lib+bin case therefore
holds for libraries without `[deps]` — the framework qualifies. Recorded
here, not fixed; a dep-aware explicit build is future work.
---
## Task 1 — the manifest `kind` key + WO-E109
**Files:**
- Modify: `compiler/bin/main.ml` (`manifest_parse` known-key table
~line 808; `manifest_build` ~line 1023).
**Interfaces:**
- Produces: `manifest_parse` accepts top-level `kind`; `manifest_build`
exposes the validated kind ("program" when absent) to Task 2's branch.
- [ ] `manifest_parse`: add `kind` to the known TOP-LEVEL keys (the match
arm that today allows `name`, `version`, `description`). Nothing else in
the parser changes — the value is a quoted string like every other key.
- [ ] `manifest_build`: after the `name` check, read `kind`. Absent or
`"program"` continues to build. `"library"` is Task 2's branch (for this
task, temporarily fall through to build — the framework does not carry
the key until Task 4, so nothing observable changes). Any OTHER value
fails as a manifest error in the WO-E106 print shape with code WO-E109,
naming the given value and the two legal ones, exit 2.
- [ ] Verify by hand: a scratch project under the scratchpad with
`kind = "junk"` refuses with WO-E109 and exit 2; the same project with
`kind = "program"`, and with no `kind` line, builds as before.
- [ ] Verify nothing moved: `just woc-test && just deps-accept && just employee`.
- [ ] Commit.
## Task 2 — library check mode + the build-error hint
**Files:**
- Modify: `compiler/bin/main.ml` (`manifest_build`; `build_mode` no-entry
error ~line 639; `usage_msg` ~lines 42–52 and the long help's build
paragraph ~line 96).
**Interfaces:**
- Consumes: Task 1's validated kind.
- Produces: `woc <dir>` on a `kind = "library"` manifest runs the full
check; the behavior Tasks 4–5 verify against.
- [ ] `manifest_build`, kind `"library"`: resolve `[deps]` and enforce the
`[runtime]` constraint exactly as build does (a library must be
checkable offline once locked), then run `compile_image ~deps dir`,
print diagnostics through the normal `finish` path on error (exit 1),
and exit 0 silently on success. No `target/` directory is created, no
file is written, no entry is required. The full pipeline runs — parse,
typecheck, interface satisfaction, ownership, GC inference — because
`compile_image` already runs it; the in-memory image is discarded
(deviation 2).
- [ ] `build_mode`, the no-entry error: when `<dir>/wo.toml` exists and
names `kind = "library"`, append the library hint to the existing
message — the project is a library; add a `main` for a demo binary or
check it with `woc <dir>`. Read the manifest only if the file exists;
malformed manifests keep failing as they do today.
- [ ] `usage_msg`: the `woc <dir>` line notes "builds a program / checks a
library, per the manifest's `kind`"; the long help gains two sentences
on `kind` and check mode. No new flags.
- [ ] Verify by hand: scratch library project (`kind = "library"`, one
class, no `main`) — `woc <dir>` exits 0 with no output; plant a type
error, `woc <dir>` exits 1 with the normal diagnostic; `woc build <dir>
-o x` fails with the no-entry message plus the library hint; add a
`main`, `woc build` produces a runnable binary (dual case) while
`woc <dir>` still only checks.
- [ ] Verify nothing moved: `just woc-test && just oop-e2e && just deps-accept`.
- [ ] Commit.
## Task 3 — the `internal/` boundary rule (WO-E108)
**Files:**
- Modify: `compiler/bin/main.ml` (`compile_image`'s dep-use walk,
~lines 487–518).
**Interfaces:**
- Consumes: the existing walk that already knows, per file, whether a dep
owns it (`owner`) and prefixes dep-internal use paths.
- Produces: WO-E108 diagnostics in the collector; Task 5 gates on the code
string.
- [ ] In the same `List.map` over parsed files: for a file NOT owned by any
dep (the `owner = None` branch, today untouched), inspect each `Use`
whose FIRST segment names a dep in `deps` — when any LATER segment is
exactly `internal`, add a diagnostic to the collector: code WO-E108, the
use's file and `pos`, message naming the internal module path and the
dependency it belongs to (the `Diag.error` + `Collector.add` pattern at
~line 277). Do not drop the use — the collector's has-error path already
prevents emission, and later resolution errors on the same use are
harmless duplicates suppressed by exit-on-first-report ordering as
today.
- [ ] Dep-owned files stay untouched on this path — the boundary is
consumer-only (spec §3): the dep's own `use internal` (prefixed to
`<dep>/internal` by the existing arm) must keep compiling.
- [ ] The root project's own `internal/` directories are NOT matched: the
rule keys on the first segment being a DEP name, so a root-project
`use internal` never fires it. No code needed — assert it in the Task 5
gate instead.
- [ ] Verify by hand: temp dir pair — a dep with an `internal/` module and
a consumer importing `<dep>/internal` — compile fails exit 1 printing
WO-E108 at the `use`'s line; the dep's own file importing `internal`
compiles.
- [ ] Verify nothing moved: `just woc-test && just deps-accept && just web-app`.
- [ ] Commit.
## Task 4 — framework reorg: adopt `kind` + `internal/`
**Files:**
- Modify: `docs/examples/writeonce-framework/wo.toml` (add
`kind = "library"`), `app.wo` (imports), `README.md` (verification note).
- Move: `http/parse.wo` → `internal/parse.wo`,
`http/serve.wo` → `internal/serve.wo` (git mv; module becomes
`framework/internal` under a consumer, `internal` standalone).
- Not touched: `http/types.wo`, `router/router.wo`, the web-app.
**Interfaces:**
- Consumes: Task 2's check mode (standalone verification), Task 3's rule
(what the move protects).
- Produces: the reorganized layout Task 5's gate checks against.
- [ ] Move the two plumbing files. Same-module access dies with the move:
`internal/serve.wo` and `internal/parse.wo` now need `use http` for
`Req`/`Resp` (they shared the `http` module with `types.wo` before);
`app.wo` adds `use internal` for the serve loop and `Parsed` seam. No
declaration changes — imports only.
- [ ] `wo.toml` gains `kind = "library"` (top-level, beside `name`).
- [ ] README: replace the `--emit` verification workaround sentence with
the check-mode invocation (`woc <dir>` — full pipeline, no entry), and
one sentence on `internal/` being unimportable by consumers.
- [ ] Verify: `compiler/_build/default/bin/woc docs/examples/writeonce-framework`
exits 0 silently — the workaround is dead.
- [ ] Verify the consumer: `just web-app` — all 14 standing checks pass
unchanged (the app imports only `framework`, `framework/http`,
`framework/router`, so NOTHING in it changes).
- [ ] Commit.
## Task 5 — gate: three new checks in web-app-accept
**Files:**
- Modify: `scripts/web-app-accept.sh` (after the build check, before the
serve matrix).
**Interfaces:**
- Consumes: the `$W/fw` framework copy and `$W/app` consumer the script
already builds; Tasks 1–4's behavior.
- Produces: `just web-app` at 17 checks, the iteration's single gate.
- [ ] Check 15 — library check mode: `woc "$W/fw"` (the framework copy,
which now carries `kind = "library"` and no `main`) exits 0 with empty
output.
- [ ] Check 16 — the boundary: copy the app to a second temp dir, append a
`use framework/internal` line to its `main.wo`, compile; assert exit 1
and `WO-E108` in stderr.
- [ ] Check 17 — kind validation: copy the app again, set
`kind = "junk"` in its manifest, compile; assert exit 2 and `WO-E109`
in stderr.
- [ ] Renumber nothing — the script counts dynamically (`pass`/`fail`);
update only the header comment's check inventory.
- [ ] Run the full battery: `just web-app` (17/0) and the standing gates —
`just woc-test`, `just oop-e2e`, `just deps-accept`, `just log-watcher`,
`just employee`.
- [ ] Commit.
## Task 6 — docs closeout
**Files:**
- Modify: `docs/00-status.md` (NEXT PLAN advances; board row 17 → done
with what landed; in-progress row moves to the next order item),
story `17-library-projects-internal.md` (landing banner),
`compiler/src/CODE-LOGIC.md` (driver section: `kind`, check mode, the
boundary rule), `docs/08-project-structure.md` (framework layout gains
`internal/`), framework story `16-web-framework.md` only if it names the
`--emit` workaround.
- [ ] Apply; every claim carries its measured gate result (17/0 etc.); no
forward-looking "will".
- [ ] `just web-app` once more after the doc edits (nothing should move —
honesty check).
- [ ] Commit.
## Success criteria (spec, restated as the gate reads them)
1. Framework checks entry-less (`woc <dir>` exit 0) and `woc build` on it
fails naming the library kind — gate check 15 + Task 2's hand check.
2. A consumer import of `framework/internal` is WO-E108 at the `use`;
the framework's own import stays legal — gate check 16 + Task 3.
3. `just web-app`'s original 14 checks pass unchanged after the reorg —
Task 4/5.
4. A manifest without `kind` behaves byte-identically — every standing
gate, every task.
## Self-review notes
- Spec coverage: §1 kind key → Task 1; §2 driver modes → Task 2; §3 rule →
Task 3; §4 diagnostics → Tasks 1–3; §5 reorg → Task 4; §6 gate → Task 5;
out-of-scope list untouched by any task. Corpus-fixture clause replaced
by deviation 1; emission-boundary wording by deviation 2; dual-with-deps
limit by deviation 3.
- Type consistency: the only cross-task names are the two code strings
(WO-E108, WO-E109), the manifest key `kind`, and the module path
`framework/internal` — spelled identically in every task.
- Risk, disclosed: moving serve/parse breaks same-module visibility they
silently enjoyed beside `types.wo`; Task 4 names the exact import each
file must gain, and the standalone check catches any miss before the
gate runs.

227
docs/superpowers/plans/2026-08-20-shard-fiber-arc.md Normal file
View file

@ -0,0 +1,227 @@
# The 8+11 concurrency arc — implementation plan (staged)
> **Status: STAGES 1 AND 2 COMPLETE 2026-08-20** (branch `concurrency-arc`,
> T1–T4 landed + the `docs/examples/fibers` demo and its `just fibers`
> gate, 8/0). Stages 2–3 pending. Execution deviations, disclosed:
> (1) the reduction budget decrements at loop BACK-EDGES ONLY, after the
> jump lands — the spec's "same three sites as the GC" wording had a
> livelock at budget 1 (pre-instruction save re-executes the jump into
> the same decrement; pinned by test_fiber's exact round-robin);
> (2) T4 landed io_uring-FIRST per the amended spec (park.c: raw ring,
> POLL_ADD/TIMEOUT, WO_IO=uring|epoll, epoll fallback) and the battery
> ran green on BOTH backends plus LW_SOAK over parked I/O;
> (3) a partial net.write's progress crosses the park via park_wr_at —
> the retry protocol needed write-offset state the spec never named.
> Stage-2 deviations: (4) the inbox is a MUTEX-guarded list + eventfd,
> not the spec's lock-free MPSC ring — rings arrive when 22 measures the
> mutex; (5) WO-E222 applies to EVERY spawn/send (round-robin placement
> makes any actor potentially remote — compile time cannot see the
> shard); (6) the deterministic corpus pins WO_SHARDS=1, and the
> multi-shard truth (output SETS, TSan, both I/O backends) lives in the
> fibers gate — the spec's narrowed-determinism rule made operational;
> (7) two TSan-caught races fixed (late-init memset vs concurrent push;
> wake-efd read outside the lock) and one teardown SEGV (routed frees
> during teardown are now no-ops: arenas die wholesale).
> Board: [docs/00-status.md](../../00-status.md).
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or
> superpowers:executing-plans task-by-task. Steps use checkbox syntax.
>
> **Style rule (user convention):** concept, reason, required behavior in
> words plus verification commands only — the executor writes the code.
**Goal:** the spec's three stages — fibers on one shard, shards with
ownership-moving sends, the transparent DB actor — each landing with
every standing gate green before the next begins.
**Architecture:** see the spec (normative):
[`../specs/2026-08-20-shard-fiber-arc-design.md`](../specs/2026-08-20-shard-fiber-arc-design.md).
Stories: [8](../../stories/language-runtime-database/refine/08-shard-actor-runtime.md) ·
[11](../../stories/language-runtime-database/refine/11-fibers.md).
**Tech Stack:** C11 libc-only (`wovm`), OCaml stdlib-only (`woc`), bash
gates; TSan added to the corpus harness at stage 2.
## Global Constraints
- Branch `concurrency-arc` off master; commits local only, never push.
- A stage does not start until the previous stage's FULL battery is
green: `just woc-test`, `just oop-e2e` (ASan stage included),
`just deps-accept`, `just web-app`, `just log-watcher`,
`just employee`.
- Stage 1 changes NO observable behavior for single-fiber programs —
every existing fixture byte-identical; that is the refactor's proof.
- New E-codes: WO-E221 (spawn needs `receive(msg: M)`), WO-E222
(traced-containing cross-shard send). `.wob` format: spawn/send lower
to new BUILTIN ids — no new opcodes; the version bumps ONLY if the
actor-type field encoding forces it (decide in T3, disclose).
---
## Stage 1 — fibers on one shard
### Task 1 — the fiber context extraction (pure refactor)
**Files:** `runtime/src/vm.h`, `runtime/src/vm.c`, `runtime/src/gc.c`
(`vm_gc_roots`), `runtime/src/main.c`, `runtime/test/*` (compile fixes
only).
**Interfaces:** produces `wo_fiber` (registers, frames, depth, catch
stack, caught error, resume pc/state) and `wo_vm` holding module + rt +
current-fiber + a fiber list; everything else consumes `vm->cur`.
- [x] Move the five interpreter-state fields into the fiber struct; the
vm allocates fiber 0 (main) at init; `vm_run`/`wo_vm_call`/unwind/
trap paths read through the current-fiber pointer.
- [x] `vm_gc_roots` iterates ALL fibers' registers+frames (one fiber
today — the loop is the change).
- [x] Verify: full battery byte-identical (this task has NO functional
change to hide behind). Commit.
### Task 2 — run queue + reduction budget
**Files:** `runtime/src/vm.c`, `runtime/src/vm.h`.
**Interfaces:** produces fiber states (RUNNABLE/PARKED/DONE), a FIFO
queue, `WO_REDUCTIONS` (default 4000), and a scheduler loop around the
dispatch loop; consumes Task 1's context switch (save/restore = swap the
current-fiber pointer).
- [x] Budget decrements — AMENDED to back-edges only (deviation 1); originally the GC-safepoint sites (NEW, CALL,
backward-JMP); zero → re-queue + switch. One fiber = no observable
change (it re-queues to itself).
- [x] A trap unwinding out of a non-main fiber kills that fiber alone
(drops run, uncaught-trap report to stderr, program lives); main's
return ends the program and unwinds every remaining fiber through the
stop path (drop-clean).
- [x] C unit test (`runtime/test/test_fiber.c`): two hand-built fibers
interleave deterministically under a tiny budget; kill-on-main-return
leaks nothing under ASan. Full battery. Commit.
### Task 3 — spawn / send / `actor M` (language + same-shard delivery)
**Files:** `compiler/src/{token,lexer,ast,parser,types,owner,emit,dump}.ml`,
`runtime/src/{wob.h,builtin.c,vm.c}`, corpus fixtures.
**Interfaces:** produces the `actor M` field-type shape (parametric like
`multi T`), `spawn Cls { ... }` expression (M inferred from Cls's
`receive`; WO-E221 otherwise), `send(addr, msg)` (msg typed M, owner
pass TRANSFERS it — sender's later use is the existing use-after-move
diagnostic), and runtime delivery: an actor is a fiber that sleeps
between messages, mailbox FIFO, one `receive` call per message.
- [x] Type/AST/parse: `actor M` in field positions; spawn/send keywords
(grep the corpus for identifier collisions first, the standing
keyword discipline).
- [x] Owner: send's message argument is a transfer (the take machinery);
spawn's ctor literal moves its field values (existing ctor rules).
- [x] Emit + runtime: two builtin ids; spawn creates the actor fiber +
returns the address as a scalar word; send enqueues + wakes.
Addresses are copyable scalars; send-to-dead = silent drop (v1,
spec'd).
- [x] Corpus: `run/actor-echo` (spawn, send, receive mutates actor
state, deterministic output), `run/actor-move` (sender's reuse after
send = compile-fail fixture, actually its own
`compile-fail/send-after-move`), `compile-fail/spawn-no-receive`
(WO-E221). ASan. Full battery. Commit.
### Task 4 — parking builtins on the shard's io_uring loop (AMENDED)
**Files:** `runtime/src/builtin.c` (net/time cases), `runtime/src/vm.c`
(the per-shard ring the scheduler idles in), `runtime/src/sysio.c`.
**Interfaces:** produces PARKED fibers keyed by ring `user_data`;
consumes Task 2's states. AMENDED 2026-08-20 (developer directive +
the linux reference project): the loop is **io_uring in readiness mode**
(POLL_ADD for fds, TIMEOUT for sleeps; resume re-executes the now-ready
builtin), with the startup probe + `WO_IO=uring|epoll` override and the
epoll path kept as the PORTABILITY fallback (seccomp'd containers deny
io_uring) — CI runs the fiber tests on both. Raw syscalls, 5.1-safe ops
only; the ring layout goes in the binding doc. 23 later rides the SAME
per-shard ring for WAL WRITE+FSYNC chains, and the ops-mode ladder
(reads/writes as ring ops with completion results) is its own follow-up
task after T4 proves the loop.
- [x] Stop composes: the stop flag wakes every parked fiber into the
unwind path (the existing stop story, per fiber).
- [x] Proven live instead of a bespoke C test: the fibers demo (part 2) + the full battery over parked I/O (log-watcher's mcp loop, web-app's matrix) on BOTH backends + LW_SOAK; originally: fiber A parked in sleep, fiber B progresses, one TID;
A resumes on time. Corpus: `run/fiber-park-net` if expressible
without a peer (loopback pair via the existing net builtins — the
log-watcher mcp fixtures prove the shape); otherwise the unit test
carries it, disclosed.
- [x] Full battery + framework/web-app gates (serve loop untouched —
single fiber per process today, degenerate path). Commit. **Stage 1
complete: board + stories note it.**
## Stage 2 — shards
### Task 5 — shard threads + per-shard runtimes
**Files:** `runtime/src/{vm.h,vm.c,main.c,obj.c,gc.c}`,
`compiler/bin/main.ml` (manifest `[runtime] shards`).
- [x] One pinned pthread per shard, each with its own full `wo_rt`;
count = cores default, `WO_SHARDS`/manifest override; N=1 must be
byte-identical to stage 1 (the escape hatch and the proof).
- [x] Program mode: main runs on shard 0; with no spawns the other
shards idle at zero cost (parked on their eventfds).
- [x] Full battery at N=1 AND at default cores (no cross-shard sends
exist yet — the threads just have to not break anything). Commit.
### Task 6 — cross-shard send + traced rejection + home-routed frees
**Files:** `runtime/src/{vm.c,obj.c,gc.c}` (MPSC rings, eventfd wake,
home-free ring), `compiler/src/types.ml` (WO-E222 via the
transitively-traced check), corpus + TSan.
- [x] Round-robin spawn placement; pointer handoff; header shard id
routes the eventual free home (drained per tick, debug-asserted).
- [x] WO-E222 at compile time for traced-containing sends (reuse the
gc_may-shaped fixpoint the inference already computes).
- [x] The actor corpus under ASan AND TSan: zero races/leaks; output-SET
assertions (cross-shard nondeterminism is honest, spec'd).
- [x] Full battery at default cores. Commit. **Stage 2 complete.**
## Stage 3 — the DB actor + closing the arc
### Task 7 — transparent DB RPC
**Files:** `runtime/src/builtin.c` (db cases marshal when not on shard
0), `database/src/` untouched (the engine never learns), `runtime/src/vm.c`
(request/reply parking).
- [ ] Non-owner DB builtins marshal statement + args to shard 0, park,
resume with materialized reply; `transaction { }` travels as one unit
(18's staged batch stays owner-side).
- [ ] `just employee` + `just web-app` at default cores, answers
byte-identical to N=1 (iteration 8's criterion 4, the arc's headline
proof). Commit.
### Task 8 — the arc's closeout
- [ ] 22's benchmark re-run (or its minimal precursor if 22 has not
landed: the employee read/write loop timed) single- vs multi-shard;
numbers recorded in the stories.
- [ ] Stories 8 + 11 landing banners; board rows; graph node classes;
framework README ledger rows that the arc unblocks (streaming etc.
stay ⏸ until their own slices — the arc UNBLOCKS, it does not build
them); CODE-LOGIC files (vm fiber model, shard/mailbox model, DB RPC).
- [ ] Full battery once more after doc edits. Commit.
## Success criteria
The spec's four, verbatim — stage 1's determinism + ASan-clean unwind,
stage 2's TSan-clean moves + WO-E222, stage 3's byte-identical
multi-shard gates + recorded 22 delta, and the language having grown
exactly `spawn`/`send`/`actor M`.
## Self-review notes
- Spec coverage: Part A → T1–T4; Part B → T5–T6; Part C → T7–T8;
diagnostics T3 (E221) / T6 (E222); out-of-scope untouched.
- The riskiest surgery (T1) is deliberately a PURE refactor with a
byte-identical battery as its only claim — functional change never
hides inside it.
- Names used consistently: `wo_fiber`, `vm->cur`, WO-E221/E222,
`WO_REDUCTIONS`, `WO_SHARDS`, `actor M`, `spawn`/`send`.

8
docs/superpowers/specs/2026-08-18-web-framework-design.md
View file

@ -7,8 +7,8 @@
library, HTTP/1.1 behind a TLS-terminating reverse proxy, and (C) the parked
HTTP/2 path. Three sub-projects; A and B are the fundable ones, C is a
recorded successor.
**Relates to:** iteration 10 (`service` blocks — this framework becomes their
lowering target, not a rival), iterations 8/9f/11 (the concurrency work h2c
**Relates to:** iteration 25 (`service` blocks — this framework becomes their
lowering target, not a rival), iterations 8/23/11 (the concurrency work h2c
waits for), `docs/plan/discarded.md` (FFI reject row — load-bearing here).
## Decisions locked during brainstorming
@ -17,7 +17,7 @@ waits for), `docs/plan/discarded.md` (FFI reject row — load-bearing here).
| --- | --- |
| TLS | **Proxy-terminated** (nginx/caddy). Browsers get TLS+ALPN+h2 from the proxy; the framework speaks HTTP/1.1 (later h2c) behind it. Zero TLS in the language or runtime. Direct-serving TLS is a separate future iteration and would be judged against the FFI/libc-only doctrine then, not now. Homegrown TLS is refused outright: a decade of side-channel and certificate-validation subtleties makes it a security liability, not a milestone. |
| Dependencies | **`wo.toml [deps]` + git fetch.** A real (mini) package manager: exact-rev git dependencies, a lockfile, a per-project cache. No registry, no semver solving. |
| HTTP/2 | **v1 is HTTP/1.1 keep-alive.** h2's payoff is multiplexing, which a single blocking thread cannot exploit; h2c lands as its own iteration after shards (8) / io_uring (9f) / fibers (11). Behind the proxy, browsers see h2 from day one regardless. |
| HTTP/2 | **v1 is HTTP/1.1 keep-alive.** h2's payoff is multiplexing, which a single blocking thread cannot exploit; h2c lands as its own iteration after shards (8) / io_uring (23) / fibers (11). Behind the proxy, browsers see h2 from day one regardless. |
| Handler model | **Structural interfaces, not closures.** The language has no function values by doctrine; a route handler is a class satisfying a `Handler` interface, dispatched by ICALL — which works on today's runtime and is checked by WO-E205. |
| Incubation | Framework is born at `docs/examples/writeonce-framework/`; the consuming app at `docs/examples/web-app/` imports it **through the `[deps]` mechanism** (a local git URL), so the whole import chain is exercised by the sample. Extraction to `github.com/shoneyj/<name>` later is a `git subtree split`, not a redesign. |
@ -136,7 +136,7 @@ routes, restart-persistence check, SIGTERM.
## C. HTTP/2 (h2c) — parked successor
After iterations 8 (shards) / 9f (io_uring) / 11 (fibers): h2c framing +
After iterations 8 (shards) / 23 (io_uring) / 11 (fibers): h2c framing +
HPACK, either natively (needs a bytes/buffer type with cheap slicing — that
type rides with this iteration, not v1) or via nghttp2-in-runtime (a doctrine
decision to re-argue then, with the TweetNaCl precedent and the libc-only

159
docs/superpowers/specs/2026-08-20-library-kind-internal-design.md Normal file
View file

@ -0,0 +1,159 @@
# Iteration 17 — library projects and dependency privacy: design
> **Status: ⏸ PARKED 2026-08-20** (developer directive: framework v1 work
> proceeds instead; spec + plan stay ready on branch `library-internal`).
> Approved before parking. Decisions were settled in
> [the iteration](../../stories/language-runtime-database/17-library-projects-internal.md)
> (four forks + impact analysis); this spec makes them buildable. The plan
> follows after review. Board: [docs/00-status.md](../../00-status.md).
>
> Per repo convention this spec carries concept, reason, and required
> behavior in words only — no implementation code.
## Goal
A project can say it is a library, and a dependency can keep modules to
itself. Concretely: `docs/examples/writeonce-framework` declares
`kind = "library"` in its `wo.toml`, `woc <dir>` on it typechecks the whole
project with no entry required (the iteration-16 `--emit` workaround is
deleted), its parser and serve-loop plumbing move under `internal/` where
the web-app cannot import them, and `just web-app` proves nothing public
broke.
## Background (the two gaps, from iterations 15/16)
1. A project without `fn main` cannot be checked: manifest presence forces
build mode, which errors "no `main` entry point found". The framework is
verified today through `woc --emit` — a wart.
2. `pub` is module-public with no dep-private tier: `parse_request` is
exactly as importable by the consuming app as `Handler`. Nothing marks
"this module is the library's own business".
## Settled decisions (normative, from the iteration)
- Library-ness is manifest-declared: top-level `kind` key, values
`"program"` (the default when absent) and `"library"`; any other value is
a manifest error.
- Privacy is Go's `internal/` directory rule, applied at the `[deps]`
boundary only: a consumer cannot `use` a dependency module whose path
contains an `internal` segment; inside the dependency the same import
stays legal.
- Lib+bin duality is allowed: a library may carry an entry-shaped `main`;
`kind = "library"` changes only the DEFAULT action of `woc <dir>`.
- Inherited from Go, explicitly not checked: an internal type may appear in
a public signature. The consumer can hold and pass such a value but
cannot import the module to name its type. Library author's smell to
avoid, not a diagnostic.
## Design
### 1. The manifest `kind` key
One top-level key in `wo.toml`, read where the manifest is already parsed
(the driver's manifest reader in `compiler/bin/main.ml`). Absent means
program — every existing project keeps its behavior. A value other than
`"program"` or `"library"` is rejected with the new WO-E109, naming the
value and the two legal ones. The key is meaningful only in the project
being invoked; a DEPENDENCY's `kind` is read but not enforced in v1
(iteration 15 already never uses a dep's `main`, so a program consumed as a
dep already behaves as a library — recorded, not policed).
### 2. Driver modes
- `woc <dir>` on a program: unchanged — build, requiring `main`.
- `woc <dir>` on a library: CHECK mode — the full pipeline runs (parse,
typecheck, interface satisfaction, borrow/ownership pass, GC inference)
over the whole project including its `[deps]`, and stops before image
emission. No entry is required. Exit 0 with no output on success, 1 with
diagnostics otherwise — a green check must mean exactly what a green
build means, minus the artifact. Deps are fetched/locked the same as
build mode (a library must be checkable offline once locked).
- `woc build <dir> -o <app>` on a library WITH a `main`: builds the binary
— the dual case (demo/self-test). Without a `main`: the existing
no-entry error, extended to name the kind so the message explains itself
("this project is a library; add a main or check it with woc <dir>").
- `--emit`, dump flags, `--update-deps`: unchanged; they never required an
entry or already carry their own rules.
### 3. The `internal/` rule
Where: the dep-use resolution step in `compile_image` — the same place
iteration 15 prefixes dep module paths — because that is the only spot that
knows which file belongs to which project. The rule: a `use` in a file that
does NOT belong to dependency X, naming a module of dependency X whose
dep-relative path contains a segment exactly equal to `internal`, is
rejected with WO-E108 at that `use`, naming the dependency and the module.
Files INSIDE dependency X importing the same module are untouched. Modules
are directory-shaped (`module_of_multi`: dep name + relative directory), so
"segment" means a path component — `framework/internal` and anything under
it, never a substring match (a module named `internals_x` is not caught).
Scope notes, normative: the rule fires only across the `[deps]` boundary
(decision 3 — dep-boundary-only; Go's subtree rule is a recorded possible
tightening). The root project's own `internal/` directories are legal to
import from anywhere inside the root project. Transitive deps stay
rejected by iteration 15's WO-E106, so dep-to-dep imports cannot occur.
### 4. Diagnostics (error catalog additions)
- **WO-E108** — dep-internal module imported across the `[deps]` boundary.
Points at the offending `use`, names the dependency and the internal
module, and says the module is internal to that dependency.
- **WO-E109** — invalid manifest `kind` value. Names the given value and
the two legal ones.
- The no-entry build error gains the library wording described in §2; no
new code, better message.
### 5. Framework reorg (the proof by use)
- `wo.toml` gains `kind = "library"`.
- `http/parse.wo` and `http/serve.wo` move to `internal/` (module
`framework/internal` when consumed). They are plumbing the web-app never
imports: the request parser, the carry-state record, the serve loop, the
`Dispatcher` seam.
- `http/types.wo` (Req/Resp + builders), `router/router.wo`
(Handler/Middleware/Route/Mw), and `app.wo` (App) stay where they are —
the public surface does not move.
- Intra-framework imports update to the new module path; the web-app
changes NOTHING — it already imports only `framework`, `framework/http`,
`framework/router`.
- The framework README's verification note replaces the `--emit`
workaround with the check-mode invocation.
### 6. Acceptance gate
Extend `scripts/web-app-accept.sh` (it already builds the framework remote
and the consuming app): the standing 14 checks stay, plus (a) check mode —
`woc` on the framework copy exits 0 with no entry present; (b) privacy — a
temp copy of the app with one added `use framework/internal` fails
compile and the output names WO-E108; (c) kind validation — a temp
manifest with a junk `kind` fails naming WO-E109. Compiler-side, the
corpus grows compile-fail fixtures for WO-E108/E109 and a run fixture is
not needed (no runtime behavior exists to pin — the VM is untouched by
design). Standing gates (`just woc-test`, `just deps-accept`,
`just oop-e2e`, samples) stay green.
## Out of scope (recorded, deliberate)
Go's full subtree rule; a `[lib]` manifest section; `pub(lib)`-style
keyword visibility; manifest export allowlists; multiple named binaries
per project (`cmd/` convention); enforcing a dep's own `kind`; internal
types in public signatures as a diagnostic; any VM, `.wob`, or GC change
(impact analysis in the iteration: visibility is compile-time name
resolution; libraries compile whole-program into the consumer's image;
GC inference stays whole-program and app usage may promote dep classes —
intended).
## Success criteria
1. **Given** the framework with `kind = "library"` and no `main`, **when**
`woc <dir>` runs, **then** it exits 0 having run the full pipeline, and
`woc build` on it fails with the message naming the kind.
2. **Given** the web-app importing `framework/internal`, **when** it
compiles, **then** WO-E108 points at the `use` and names the framework;
the framework's own files importing it stay legal.
3. **Given** the reorganized framework, **when** `just web-app` runs,
**then** all standing checks pass unchanged — the public surface did
not move.
4. **Given** any existing project with no `kind` key, **when** it builds,
**then** nothing changed — absent means program.

243
docs/superpowers/specs/2026-08-20-shard-fiber-arc-design.md Normal file
View file

@ -0,0 +1,243 @@
# The 8+11 concurrency arc — shards, fibers, actors: design
> **Status: spec, awaiting review (2026-08-20).** The arc's decisions were
> settled in [iteration 8](../../stories/language-runtime-database/refine/08-shard-actor-runtime.md)
> (refined) and the brainstorm of 2026-08-20 (this document's Decisions).
> Covers iterations 8 AND 11 as one arc; iteration 24 (chat) is its
> acceptance workload and gets its own spec after this one. The plan
> follows after review. Board: [docs/00-status.md](../../00-status.md).
>
> Per repo convention: concept, reason, and required behavior in words
> only — no implementation code.
## Goal
The runtime scales past one core without ever showing a thread, a lock,
or a colored function: pinned thread-per-core shards whose only
communication is ownership-moving messages, cooperative fibers above them
preempted by reduction budget, blocking builtins that park instead of
block, and a database that stays single-writer by being runtime plumbing
on its owner shard. One `spawn`/`send` surface; app code byte-identical
at one shard or sixteen.
## Decisions (settled; the spec builds on them)
From the 2026-08-20 refinement: the DB is an actor on an owner shard
(never user-visible); 8+11 are one arc; one unified address surface;
chat (19) is the driving workload; order 22 → arc → 19 → 23.
From the arc brainstorm (2026-08-20):
1. **All cores by default.** `WO_SHARDS` / `[runtime] shards = N`
override (N=1 is the debug/serial escape hatch); the arc's landing
criterion is every standing gate green AT the default. Brave by
choice: the gates become the stress test.
2. **Typed receive.** An actor is any class with `fn receive(msg: M)` —
structural, the Handler doctrine, no blocking-receive keyword, no new
interface to implement. `spawn Cls { ... }` infers M from the class's
`receive` signature and returns an **`actor M`** address (a
parametric builtin type exactly like `multi T`/`map<K, V>`).
`send(addr, msg)` compile-checks that msg is an M and MOVES it.
3. **Transparent DB RPC.** insert/update/delete/query builtins executed
on a non-owner shard marshal the statement to the owner shard and
PARK the calling fiber until the reply; results come back
materialized (they already do). App code never changes; the fourth
iteration-8 criterion (byte-identical answers single- or multi-shard)
is the proof.
4. **Fibers first, then shards.** Stage 1 ships fibers on one shard
(framework's fiber-per-connection lands early, no threads); stage 2
ships pinned threads + cross-shard send; stage 3 ships the DB RPC and
re-runs 22. Each stage gates green independently.
## Part A — the fiber runtime (stage 1, one shard)
### The fiber context
A fiber IS the interpreter state `wo_vm` already isolates: the register
window (`WO_STACK_SLOTS`), the frame stack (`WO_MAX_FRAMES`), the catch
stack, the caught-error slot, and a resume point. Stage 1 extracts those
fields into a fiber context; `wo_vm` keeps the module, the runtime
(arena/GC/db — per SHARD, not per fiber), the current-fiber pointer, and
a FIFO run queue. Cost, stated: a context is ~42 KiB at today's
constants (32 KiB registers + frames + catches), so 1k fibers ≈ 42 MiB —
acceptable v1, arena-allocated; segmented/growable windows are recorded
future work, not v1.
### Scheduling
- FIFO run queue, no priorities (iteration 11's out-of-scope holds).
- **Preemption by reduction budget**: the budget decrements at the
EXISTING safepoint sites (NEW, CALL, backward-JMP — the same places
7b's GC polls, one counter check added, no new instrumentation); at
zero the fiber re-queues and the next runs. Default budget 4000
reductions, `WO_REDUCTIONS` overrides. Deterministic: same program,
same schedule.
- `main` is fiber 0. When it returns, the program's exit value is
main's; every other fiber — runnable or parked — is UNWOUND through
the drop machinery (the SIGTERM/stop story, reused verbatim): ASan
zero leaks is the criterion. A trap that unwinds out of a non-main
fiber kills that fiber alone (its drops run, its error goes to stderr
the uncaught-trap way); the program does not die.
- GC: every fiber's registers and frames are roots — `vm_gc_roots`
iterates all contexts, not just the live one. Parked and queued
fibers pin their values exactly like the running one.
### spawn / send (same-shard in stage 1)
- `spawn Cls { fields }` — an expression; Cls must declare
`fn receive(msg: M)` where M is a class, record, or union type
(**WO-E221** otherwise, naming what's missing); the ctor literal's
fields are the actor's state, moved in (the take-push shape). Result
type `actor M`.
- `send(addr, msg)`: msg must type as M (existing assign/param
machinery) and be OWNED (the owner pass's transfer rules; a borrowed
or copy-in-hand value is the existing WO-E3xx shape). After the send
the sender's binding is dead — compile-time move, the iteration-8
criterion.
- Delivery: the runtime calls the actee's `receive` with the message,
one message at a time per actor, on the actor's home shard, as its
fiber's next work item. An actor is a fiber that sleeps between
messages; its mailbox is the FIFO the runtime owns.
- Same-heap sends hand the pointer over — no copy, no serialization;
the move made aliasing impossible at compile time.
- `actor M` values are plain copyable words (an address), storable in
fields/containers like Int — sending TO a dead actor is a silent drop
of the message v1 (Erlang's shape; a delivery-receipt story is future
work, recorded).
### The I/O plane: io_uring-first (AMENDED 2026-08-20, developer directive)
The original wording made epoll the v1 loop with io_uring as a
benchmark-gated swap. That framing is REVERSED by directive, and the
[linux reference project](../../plan/exploration/linux/07-io_uring.md)
already pointed here: io_uring is the successor to epoll + libaio for
BOTH the storage engine's WAL path and the socket path — one event loop.
- **One io_uring per shard thread** is THE event loop, serving three op
families as the arc progresses: (1) fiber parking — `IORING_OP_POLL_ADD`
for parked net fds, `IORING_OP_TIMEOUT` for sleeps, the mailbox eventfd
registered the same way (stage 2's wake); (2) WAL durability — 23's
WRITE + FSYNC(DATASYNC) chains ride the SAME ring, `user_data` the
commit correlation, the shard executing while the ring drains; (3) the
data path — multishot accept/recv/send for iteration 24's connection
fan-in, LAST, because it changes resume semantics.
- **The readiness→ops ladder, staged honestly:** T4 uses the ring in
READINESS mode (POLL_ADD/TIMEOUT — resume re-executes the now-ready
builtin, the same retry contract epoll would have had, so the builtin
layer barely changes). Submitting the reads/writes THEMSELVES as ring
ops (completion carries the result, no retry) is the follow-up step —
bigger builtin surgery, staged after T4 proves the ring loop.
- **Kernel floor and ops discipline:** io_uring is Linux 5.1+, mature
5.11+; v1 uses only 5.1-safe ops (POLL_ADD, TIMEOUT, WRITE, FSYNC).
Multishot accept (5.19) / multishot recv (6.0) are recorded
optimizations for 19's spec, gated on the probe, never assumed.
- **The fallback is PORTABILITY, not preference:** a startup probe
(`io_uring_setup`, fall back on ENOSYS/EPERM — seccomp'd containers
routinely deny io_uring) selects the ring or the epoll+blocking-fsync
path; `WO_IO=uring|epoll` overrides so CI proves BOTH paths on one
kernel (this subsumes 23's `WO_WAL_MODE`). io_uring is the design;
epoll exists so the binary runs everywhere — the project's premise.
- Raw `io_uring_setup`/`io_uring_enter` syscalls, libc-only (23's settled
fork, now arc-wide); the mmap'd ring layout goes in the binding doc,
normative, exactly like the WAL format.
- `net.accept`/`net.read`/`net.write` and `time.sleep` PARK the calling
fiber against the ring; the shard runs other fibers; completion
re-queues the parked one. With exactly one fiber the ring wait IS the
blocking call — same code path, program mode is the one-fiber case.
- `fs.*` stays genuinely blocking in v1 (local disk, bounded); its reads
become ring ops in the ops stage if 22's numbers ask; recorded.
- The stop story composes: the stop signal wakes the ring (the existing
self-pipe/eventfd trick), every parked fiber unwinds — the SIGTERM
drain criterion extends to fibers for free.
- The framework consequence (its own slice, after stage 1): the serve
loop spawns a fiber per connection and the close-when-idle policy
dies; that lands with iteration 24's spec, not this one.
### Fiber context growth (recorded improvement, post-stage-1)
A fiber context is ~42 KiB today (fixed register window + frame array).
Unlike native-stack green threads (Go's grow-by-copy must rewrite every
pointer into the old stack), this VM's "stack" is register-INDEXED, not
address-based — a context is relocatable by construction, so growth is
an allocate-larger + memcpy with zero pointer fixups. Start-small
(~4 KiB) growable contexts are therefore cheap to add and take fiber
counts from thousands toward millions; scheduled after stage 1, before
iteration 24's 1k-connection target if measurement asks.
## Part B — shards (stage 2)
- One pinned pthread per shard; shard count = cores by default
(`WO_SHARDS`/manifest override). Each shard owns a full `wo_rt`:
arena, traced list, GC — 7b's collector is per-shard by construction,
so no global pause exists to remove.
- Cross-shard `send`: an MPSC mailbox ring per shard plus an eventfd to
wake an idle shard's epoll loop (the c-runtime exploration's proven
pair). The message POINTER crosses; the object's home shard is stamped
in its header (there since iteration 2); its eventual free routes back
to the allocation-home arena (a small home-free ring per shard,
drained at the tick).
- What may cross: OWNED values whose class transitively contains no
TRACED field — the `gc_may`-shaped fixpoint the runtime already
computes, surfaced at compile time: sending a value whose type is or
contains an inferred-traced class is **WO-E222**, naming the traced
class and why it is traced. Aliased graphs never cross heaps.
- `spawn` placement: round-robin across shards by default; no placement
argument in v1 (recorded future work if 22 shows a need). The
language never names a shard.
- Determinism criterion narrows honestly: single-shard runs stay
deterministic (stage 1's property); cross-shard interleaving is
nondeterministic by nature — the corpus asserts OUTPUT SETS and
ownership invariants under TSan, not byte-identical transcripts.
## Part C — the DB actor + serving (stage 3)
- The engine and WAL live on shard 0 (the owner). DB builtins executed
on the owner run exactly today's code. On any other shard they
marshal the statement into a runtime message, park the fiber, and
resume with the materialized reply. `transaction { }` (iteration 18)
marshals as one unit — the staged batch stays owner-side, semantics
unchanged.
- The listener: `net.listen` + `net.accept` stay on the shard that
calls them v1 (the framework serves from one accept loop; accepted
connections' fibers stay on that shard). Distributing accept
(SO_REUSEPORT per shard) is 19/23-adjacent future work the benchmark
must justify — recorded, not built.
- Stage 3 closes the arc: `just employee`/`just web-app` run
multi-shard byte-identical (criterion 4), the actor corpus runs under
ASan+TSan, and 22 re-runs with the before/after recorded.
## Diagnostics (new)
- **WO-E221** — `spawn` on a class with no `receive(msg: M)` (or an M
that is not a class/record/union).
- **WO-E222** — cross-shard send of a traced (or traced-containing)
type; names the class and the inference reason. Compile-time; stage 1
same-shard sends are exempt (same heap, aliasing is GC's problem and
GC handles it).
- Existing machinery covers the rest: send-of-borrowed is the owner
pass's transfer rules; send-type mismatch is the parameter boundary.
## Out of scope (arc-wide, restated)
Fiber migration across shards; priorities/timers/structured concurrency
(recipe-box, later `.wo` libraries); async/await (permanently rejected);
cross-shard transactions/2PC; SO_REUSEPORT accept distribution;
segmented fiber stacks; delivery receipts / actor supervision trees
(future story); program-mode API changes (none — one fiber is the
degenerate case).
## Success criteria
1. **Stage 1:** thousands of fibers on one shard, one hot loop among
them — everything progresses (starvation-free fixture), output
deterministic; a parked-fiber shutdown unwinds ASan-clean; a fiber
blocked in `net.read` resumes while the shard served others, one TID.
2. **Stage 2:** the actor corpus under ASan+TSan — zero races, zero
leaks; an owned send compiles as a move (sender's later use is a
compile error); a traced-containing send is WO-E222; frees route
home (asserted in debug builds).
3. **Stage 3:** `just employee` and `just web-app` green at the
all-cores DEFAULT with byte-identical answers; 22's benchmark re-run
with the delta recorded; every standing gate green.
4. The language grew exactly `spawn`, `send`, and the `actor M` type —
no async/await, no locks, no thread ever visible.