diff --git a/docs/00-code-review.md b/docs/00-code-review.md index abb586a..e380372 100644 --- a/docs/00-code-review.md +++ b/docs/00-code-review.md @@ -12,4 +12,4 @@ Native speed — the big one. Everything is interpreted: ~40× behind Go on raw - Language expressiveness. No generics — the cache stores Text and tells you to json.encode; no function values or closures (doctrine, but it's why every handler is a class with one method); byte-based strings with no Unicode awareness; no Result-style error values (traps + try only); pattern matching is a switch, not destructuring. Some of this is deliberate rejection, but "deliberate" doesn't make the expressiveness appear. - Concurrency holes the arc hasn't closed. send is one-way — no reply/request-response primitive (my own benchmarks couldn't await the actor and had to sleep); no supervision, links, or actor death (actors live until process end); unbounded mailboxes with zero backpressure; no timers beyond sleep; round-robin placement with no work stealing; multi-shard DB access still traps (stage 3 unbuilt); accept lives on one shard. - Production plumbing. No TLS anywhere (proxy-mandated forever), no HTTP/2 or WebSockets yet, no crypto primitives (blocked on the bit-ops-vs-builtins fork), observability is print/stderr — no metrics, tracing, or profiler; no debugger, no LSP (discussed, never built); deps are git-rev-only with no registry, no transitive resolution, no semver; blue-green deploy and schema migrations are recorded futures, not features. -- Proof maturity. 9e's benchmark battery has never run — every number so far is a scratch measurement on one machine; TSan covers one demo; no fuzzing, no CI beyond local just, and the whole ecosystem is one framework, five samples, and one committed consumer. The honest summary: the architecture is ahead of the product — the doctrine bets (ownership+inference, actors, one binary, io_uring) are landing and measurable, while the surface a developer touches daily (types, tooling, ecosystem) is years behind the languages it benchmarks against. +- Proof maturity. 22's benchmark battery has never run — every number so far is a scratch measurement on one machine; TSan covers one demo; no fuzzing, no CI beyond local just, and the whole ecosystem is one framework, five samples, and one committed consumer. The honest summary: the architecture is ahead of the product — the doctrine bets (ownership+inference, actors, one binary, io_uring) are landing and measurable, while the surface a developer touches daily (types, tooling, ecosystem) is years behind the languages it benchmarks against. diff --git a/docs/00-dependency-graph.md b/docs/00-dependency-graph.md index 9ab8fd4..850e701 100644 --- a/docs/00-dependency-graph.md +++ b/docs/00-dependency-graph.md @@ -7,7 +7,7 @@ > green is startable today. Rebuilt 2026-08-20 from a sweep of every > story/spec/plan markdown (the "misses" pass: iteration 17's outgoing > edges, the concurrency chain, the post-12 parked drain, 9b→10, -> 14's gap fan-out, 9c's fiber caveat). +> 14's gap fan-out, 20's fiber caveat). ## 1. Story iterations @@ -30,17 +30,17 @@ flowchart TD FWREORG["framework internal/ reorg + check mode (kills the --emit workaround; WO-E108/E109 reserved)"]:::parked I18["18 framework v2: transaction{} + cache/flags/jobs (spec APPROVED — the next implementation)"]:::specd - I9c["9c cross-program tables (half-built)"]:::open - I9d["9d keypair attach auth (half-built; crypto+handshake already on its branch)"]:::open - I9e["9e durability + throughput baseline"]:::open + I9c["20 cross-program tables (half-built)"]:::open + I9d["21 keypair attach auth (half-built; crypto+handshake already on its branch)"]:::open + I9e["22 durability + throughput baseline"]:::open I8["8 shard-actor runtime"]:::open - I9f["9f io_uring group-commit"]:::open + I9f["23 io_uring group-commit"]:::open I10["10 HTTP service layer (lowers onto the framework)"]:::open I11["11 fibers"]:::open I12["12 blue-green deploy"]:::open I13["13 metaprogramming @derive"]:::open I14["14 skillhost workload (demoted)"]:::open - I9g["9g query grammar corpus (likely collapses)"]:::open + I9g["27 query grammar corpus (likely collapses)"]:::open GAPS["14's gap fan-out: bounded subprocess, stdin/stdout transport, fs metadata, FFI-vs-out-of-process"]:::open DRAIN["post-12 parked drain: pub(read)/using/#if, WO-E225, ADT roster, group-by"]:::parked @@ -75,14 +75,14 @@ flowchart TD ``` Reading it: **18 is the only spec-approved open node with all -prerequisites green — the next implementation.** After 18: 9c/9d and 9e -are startable (chosen order: 9c/9d first — half-built branches rot). +prerequisites green — the next implementation.** After 18: 20/21 and 22 +are startable (chosen order: 20/21 first — half-built branches rot). 17 unparks on directive: its prerequisites landed, its spec+plan wait on branch `library-internal`, and its landing brings the framework reorg node with it. 13 and the parked drain sit behind 12 by the 2026-08-08 scope directive (dashed), not by any technical edge. -## 2. The concurrency chain (iterations 8 / 9f / 11 and everything they gate) +## 2. The concurrency chain (iterations 8 / 23 / 11 and everything they gate) The runtime's concurrency work is the single biggest unlocker — every ⏸ row in the framework ledger and two v2 follow-ons hang off it. @@ -95,17 +95,17 @@ flowchart TD I7b2["7b per-shard collector (done — the precondition 8 waited on)"]:::rt I8x["8 shard-actor runtime: thread-per-core, ownership-move messages"]:::rt - I9fx["9f io_uring group-commit (batch = the shard tick)"]:::rt + I9fx["23 io_uring group-commit (batch = the shard tick)"]:::rt I11x["11 fibers: reduction-budget preemption, blocking builtins park"]:::rt - I9ex["9e baseline (numbers 8/9f sign against)"]:::rt + I9ex["22 baseline (numbers 8/23 sign against)"]:::rt KEEPAL["keep-alive parking retired (close-when-idle policy dies; parked fds)"]:::gated - H2C2["h2c HTTP/2 cleartext (spec §C: also needs 9f)"]:::gated + H2C2["h2c HTTP/2 cleartext (spec §C: also needs 23)"]:::gated STREAM2["request body streaming + backpressure"]:::gated SRESP2["streaming responses + explicit commit point"]:::gated CANCEL2["per-request cancellation propagation"]:::gated PUBSUB2["pub/sub + WebSockets (rejected until here)"]:::gated - ASYNC9C["9c async attach statements (rejected-for-now alternative)"]:::gated + ASYNC9C["20 async attach statements (rejected-for-now alternative)"]:::gated TIMEOUTS2["idle timeouts become schedulable (net seam still needed)"]:::gated FIBJOBS2["fiber-scheduled jobs (replaces drain-on-request; queue table stays)"]:::v2 @@ -164,7 +164,7 @@ flowchart TD UNIX["unix socket binding"]:::blocked PEERV["trusted-proxy PEER verification"]:::blocked - CRYPTO["GATE: crypto fork — C builtins vs language bit ops (brainstorm); digests want iteration 20's Bytes"]:::gate + CRYPTO["GATE: crypto fork — C builtins vs language bit ops (brainstorm); digests want iteration 19's Bytes"]:::gate SHA["SHA-256/512, HMAC, CRC32"]:::blocked ETAG["ETag + conditional requests"]:::blocked COOKIE["signed cookies"]:::blocked @@ -174,7 +174,7 @@ flowchart TD JWT["JWT HS256 (HARD STOP after)"]:::blocked RADIX["radix-tree routing"]:::blocked - I9E3["GATE: 9e measures the linear scan"]:::gate + I9E3["GATE: 22 measures the linear scan"]:::gate STORAGE["storage-integration rows: migrations (future story), eager loading + tenant roots (query-surface work, 9-series)"]:::blocked @@ -195,7 +195,7 @@ flowchart TD Green nodes (CORS, security headers, host validation, strict-parsing audit, wildcards, route groups, `req.ctx`, XFF parsing, Accept negotiation) need nothing — startable in any order, gated by -`just web-app`. Note: 9d's keypair crypto is its own C implementation +`just web-app`. Note: 21's keypair crypto is its own C implementation (already on branch `keypair-auth`) — it neither waits for nor feeds the crypto-fork gate. diff --git a/docs/00-status.md b/docs/00-status.md index 81f642f..a409ba2 100644 --- a/docs/00-status.md +++ b/docs/00-status.md @@ -68,7 +68,7 @@ operators, streaming/cancellation park behind 8/11). The memory-rich features are **framework v2** = iteration 18 (spec APPROVED 2026-08-20, plan next): TTL cache, @table flags, durable job queue with drain-on-request, `transaction { }` over the WAL's staged batch. After 18, -the order resumes at 9c/9d. Edges: [00-dependency-graph.md](00-dependency-graph.md). +the order resumes at 20/21. Edges: [00-dependency-graph.md](00-dependency-graph.md). --- @@ -171,18 +171,18 @@ that sequences its tasks. Read one, approve, then the next starts. | 8 | [Shard-actor runtime](stories/language-runtime-database/08-shard-actor-runtime.md) | ⬜ | | 9 | [Database engine](stories/language-runtime-database/done/09-database-engine.md) | 🔄 engine complete (storage/WAL/indexes/insert-update-delete); reads land with 9b | | 9b | [`@table`, relations, query](stories/language-runtime-database/done/09b-table-relations-query.md) | 🔄 query surface + relations + FK done (branch query-surface); group-by parked | -| 9c | [Cross-program tables](stories/language-runtime-database/refine/09c-cross-program-tables.md) | 🔄 channel done (branch ipc-attach); manifest+binding pending | -| 9d | [Keypair attach auth](stories/language-runtime-database/refine/09d-keypair-attach-auth.md) | 🔄 crypto+handshake done (branch keypair-auth); manifest pending | -| 9e | [Durability, throughput, scale](stories/language-runtime-database/refine/09e-durability-throughput-scale.md) | ⬜ needs a spec first | -| 9f | [io_uring group-commit](stories/language-runtime-database/refine/09f-io-uring-commit.md) | ⬜ after 8 + 9e | -| 9g | [Query grammar corpus](stories/language-runtime-database/refine/09g-query-grammar-corpus.md) | ⬜ needs a spec first | -| 10 | [HTTP service layer](stories/language-runtime-database/10-http-service.md) | ⬜ | Hold | +| 20 | [Cross-program tables](stories/language-runtime-database/refine/20-cross-program-tables.md) | 🔄 channel done (branch ipc-attach); manifest+binding pending | +| 21 | [Keypair attach auth](stories/language-runtime-database/refine/21-keypair-attach-auth.md) | 🔄 crypto+handshake done (branch keypair-auth); manifest pending | +| 22 | [Durability, throughput, scale](stories/language-runtime-database/refine/22-durability-throughput-scale.md) | ⬜ needs a spec first | +| 23 | [io_uring group-commit](stories/language-runtime-database/refine/23-io-uring-commit.md) | ⬜ after 8 + 22 | +| 27 | [Query grammar corpus](stories/language-runtime-database/refine/27-query-grammar-corpus.md) | ⬜ needs a spec first | +| 10 | [HTTP service layer](stories/language-runtime-database/25-http-service.md) | ⬜ | Hold | | 11 | [Fibers](stories/language-runtime-database/refine/11-fibers.md) | ⬜ | Hold | -| 12 | [Blue-green deploy](stories/language-runtime-database/12-blue-green-deploy.md) | ⬜ | Hold | -| 13 | [Compile-time metaprogramming](stories/language-runtime-database/refine/13-compile-time-metaprogramming.md) | ⬜ needs a spec first | -| 14 | [skillhost host workload](stories/language-runtime-database/refine/14-skillhost-host-workload.md) | ⬜ gaps recorded (branch query-grammar found skillhost needs no new query grammar); each gap a candidate iteration | +| 12 | [Blue-green deploy](stories/language-runtime-database/26-blue-green-deploy.md) | ⬜ | Hold | +| 13 | [Compile-time metaprogramming](stories/language-runtime-database/refine/29-compile-time-metaprogramming.md) | ⬜ needs a spec first | +| 14 | [skillhost host workload](stories/language-runtime-database/refine/28-skillhost-host-workload.md) | ⬜ gaps recorded (branch query-grammar found skillhost needs no new query grammar); each gap a candidate iteration | | 15 | [deps: `wo.toml [deps]`](stories/language-runtime-database/done/15-deps-package-manager.md) | ✅ **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 | -| 16 | [web framework](stories/language-runtime-database/done/16-web-framework.md) | ✅ **landed 2026-08-19** — writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (§C) behind 8/9f/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 | +| 16 | [web framework](stories/language-runtime-database/done/16-web-framework.md) | ✅ **landed 2026-08-19** — writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (§C) behind 8/23/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 | | 17 | [library projects + `internal/`](stories/language-runtime-database/17-library-projects-internal.md) | ⏸ **PARKED 2026-08-20** (developer directive; framework v1 first) — forks settled, spec + plan approved and ready on branch `library-internal`: kind = "library" key; Go internal/ rule, dep-boundary-only; lib+bin dual; VM/GC untouched by design | | 18 | [framework v2: memory-rich features](stories/language-runtime-database/18-memory-db-features.md) | 🔄 **spec APPROVED 2026-08-20, plan next** ([spec](superpowers/specs/2026-08-20-memory-db-features-design.md)): TTL cache + @table flags + durable job queue (drain-on-request) + `transaction { }` over the WAL's staged batch; pub/sub REJECTED until 8/11 | @@ -192,7 +192,7 @@ that sequences its tasks. Read one, approve, then the next starts. | Track | Item | Where | | -------- | --------------------------------------------------------------------------- | ---------------------------------------------------------- | -| Language | nothing active — the framework v1-polish slice landed 2026-08-20 (branch framework-v1, awaiting merge); next per the order: brainstorm 9c/9d's forks | [order](#implementation-order-re-sequenced-2026-08-20--framework-goal) | +| Language | nothing active — the framework v1-polish slice landed 2026-08-20 (branch framework-v1, awaiting merge); next per the order: brainstorm 20/21's forks | [order](#implementation-order-re-sequenced-2026-08-20--framework-goal) | Off-goal work is parked; the goal (2026-08-20) is the web framework as a polished micro-framework v1 — iteration 17 (library kind + `internal/`) is @@ -362,11 +362,11 @@ The C proving-ground work (`exploration/c-runtime/`, phases A–F: 859k reads/s, ### Implementation order (re-sequenced 2026-08-20 — framework goal) The goal is the web framework as a first-class library, so the framework -line leads and the workload-driven extras (14, 9g) demote behind it. -Dependency rules that force the shape: 9f explicitly after 8 + 9e; 9c -"precedes iteration 10"; 9d's plan folds into 9c's; 12 only after 9 + 10 +line leads and the workload-driven extras (14, 27) demote behind it. +Dependency rules that force the shape: 23 explicitly after 8 + 22; 20 +"precedes iteration 25"; 21's plan folds into 20's; 12 only after 9 + 10 (catalog to diff, HTTP to build on); 11 rides 8's shard scheduler; h2c -parked behind 8/9f/11; the post-12 parked list stays parked by the +parked behind 8/23/11; the post-12 parked list stays parked by the 2026-08-08 scope directive. (Iteration 7 dropped from this list — landed 2026-08-15.) @@ -379,27 +379,27 @@ parked behind 8/9f/11; the post-12 parked list stays parked by the APPROVED 2026-08-20, the only all-green spec-approved node in the [dependency graph](00-dependency-graph.md) — plan next, then implement. -3. **9c then 9d** — finish the half-done branches (ipc-attach: manifest + - binding; keypair-auth: manifest) before they rot; 9d folds into 9c's +3. **20 then 21** — finish the half-done branches (ipc-attach: manifest + + binding; keypair-auth: manifest) before they rot; 21 folds into 20's plan; both must precede 10. -4. **9e** — the measurement backbone; baseline single-shard BEFORE the - runtime restructure so 8/9f/11 sign against real numbers. +4. **22** — the measurement backbone; baseline single-shard BEFORE the + runtime restructure so 8/23/11 sign against real numbers. 5. **8** — shard-actor; the framework's multi-core serving story; unblocks - 9f, 11, h2c. -6. **9f** — io_uring group-commit; explicitly after 8 + 9e. + 23, 11, h2c. +6. **23** — io_uring group-commit; explicitly after 8 + 22. 7. **11** — fibers; retires the framework's disclosed keep-alive limit (an idle connection starving accept forced close-when-idle in iteration 16 — - parked fds fix it properly); with 8 + 9f done, **h2c unparks** (spec §C) + parked fds fix it properly); with 8 + 23 done, **h2c unparks** (spec §C) as the framework's HTTP/2 slice. -8. **10** — HTTP service layer; after 9c by its own precedence note; +8. **10** — HTTP service layer; after 20 by its own precedence note; `service` blocks lower onto the framework instead of a parallel stack. 9. **12** — blue-green; prerequisites 9 + 10 now exist; completes the framework's deploy story. -10. **9g then 14** — demoted with the goal shift: skillhost is no longer the - driving workload; 9g likely collapses to "confirm `len(query)` + add - `exists`" and precedes 14 when they run. +10. **27 then 14** — demoted with the goal shift: skillhost (28) is no longer the + driving workload; 27 likely collapses to "confirm `len(query)` + add + `exists`" and precedes 28 when they run. 11. **13 + parked drain** — metaprogramming (spec first), then group-by, - `pub(read)`/`using`/`#if`, ADT roster, WO-E225 — held behind 12 by the + `pub(read)`/`using`/`#if`, ADT roster, WO-E225 — held behind 26 by the scope directive. ### Language track — sequenced, on the critical path @@ -412,18 +412,18 @@ parked behind 8/9f/11; the post-12 parked list stays parked by the | 8 | Shard-actor runtime | [plan 4](superpowers/plans/2026-08-01-shard-actor-vm-runtime.md) | | 9 | Database engine binding | [plan 5](superpowers/plans/2026-08-01-db-engine-binding.md) | | 9b | `@table` + relations + language-integrated query — comprehension queries, `ref`/`backlink` navigation, GroupBy aggregates; acceptance: new `docs/examples/employee` sample | [spec](superpowers/specs/2026-08-15-table-relations-query-design.md) · [plan](plan/compiler/2026-08-15-employee-relations-query.md) | -| 9c | Cross-program tables — attach to a running program's database (IPC string in wo.toml, manifest-granted rights, owner stays the single writer) | **no spec yet** — four open forks recorded in the iteration; brainstorm before planning | -| 9d | Keypair attach auth — mutual challenge–response, grants name public keys, uid superseded | **no spec yet** — four forks recorded; plan folds into 9c's | -| 9e | Durability + throughput + scale — restart-persistence, read/write benchmark, ~1M rows; the gate every later optimization re-runs | **no spec yet** — four forks recorded; the measurement backbone | -| 9f | io_uring group-commit write path — batched durability overlapped on shard threads, fsync fallback | **no spec yet** — brainstorm after iterations 8 + 9e | -| 9g | Query grammar from real embedded-DB corpora — whole-query count + correlated exists, driven by the skillhost SQL catalogue; add only what a corpus uses | **no spec yet** — three forks; may collapse to "confirm len(query) + add exists" | +| 20 | Cross-program tables — attach to a running program's database (IPC string in wo.toml, manifest-granted rights, owner stays the single writer) | **no spec yet** — four open forks recorded in the iteration; brainstorm before planning | +| 21 | Keypair attach auth — mutual challenge–response, grants name public keys, uid superseded | **no spec yet** — four forks recorded; plan folds into 20's | +| 22 | Durability + throughput + scale — restart-persistence, read/write benchmark, ~1M rows; the gate every later optimization re-runs | **no spec yet** — four forks recorded; the measurement backbone | +| 23 | io_uring group-commit write path — batched durability overlapped on shard threads, fsync fallback | **no spec yet** — brainstorm after iterations 8 + 22 | +| 27 | Query grammar from real embedded-DB corpora — whole-query count + correlated exists, driven by the skillhost SQL catalogue; add only what a corpus uses | **no spec yet** — three forks; may collapse to "confirm len(query) + add exists" | | 14 | skillhost host workload — port skillhost (MCP host + confined script runner) to writeonce; drives the missing host capabilities into the open (bounded subprocess, stdin/stdout transport, fs metadata, FFI-vs-out-of-process) | **no spec yet** — gaps recorded in the iteration; each gap brainstormed on demand, bounded-subprocess first | | 17 | library projects + dependency privacy — `wo.toml` kind = "library" (checkable without entry, dual lib+bin) + Go-style `internal/` at the [deps] boundary; framework reorg demonstrates both | **forks settled 2026-08-20** — decisions + framework/compiler/VM/GC impact in the iteration; spec/plan next | | 10 | HTTP service layer | [plan 6](superpowers/plans/2026-08-01-http-service-layer.md) | | 11 | Fibers | vision §3, [blue-green exploration](plan/exploration/blue-green-vm/00-vision.md) | -| 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9–10 | +| 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9 + 25 | -### Language track — parked until after iteration 12 +### Language track — parked until after iteration 26 Recorded 2026-08-08 by scope directive; nothing here lands before the log-watcher proof. diff --git a/docs/examples/employee-list/README.md b/docs/examples/employee-list/README.md index 416d41a..b3ca75a 100644 --- a/docs/examples/employee-list/README.md +++ b/docs/examples/employee-list/README.md @@ -2,8 +2,8 @@ > **Status: target workload — does not compile on today's toolchain.** > Written ahead of iterations -> [9c (cross-program tables)](../../stories/language-runtime-database/refine/09c-cross-program-tables.md) -> and [9d (keypair attach auth)](../../stories/language-runtime-database/refine/09d-keypair-attach-auth.md), +> [20 (cross-program tables)](../../stories/language-runtime-database/refine/20-cross-program-tables.md) +> and [21 (keypair attach auth)](../../stories/language-runtime-database/refine/21-keypair-attach-auth.md), > the way every acceptance sample here precedes its features. It also leans > on 9/9b (the [employee sample](../employee/) it attaches to must run > first). @@ -37,6 +37,6 @@ the source says `employee.Employee`. | `employee-list staff ` | unique-name index probe + `staff` backlink scan, both in A | | `employee-list probe-write` | the rights matrix: registered read-only, so the insert traps with access-denied (caught, `DENIED …`, exit 4) and A's row count is unchanged | -The 9d acceptance drives the rest from the outside: wrong key, no key, +The 21 acceptance drives the rest from the outside: wrong key, no key, same-uid-wrong-key, impostor socket, handshake replay, key rotation — see the iteration's criteria; this sample is the workload they run against. diff --git a/docs/examples/writeonce-framework/README.md b/docs/examples/writeonce-framework/README.md index fdcdf26..bf75374 100644 --- a/docs/examples/writeonce-framework/README.md +++ b/docs/examples/writeonce-framework/README.md @@ -87,7 +87,7 @@ first (pure `.wo` cannot express it yet). | Item | State | | --- | --- | -| Path matching | 🔶 linear scan, first-match-wins; a radix tree is a performance slice that waits for iteration 9e to MEASURE it first | +| Path matching | 🔶 linear scan, first-match-wins; a radix tree is a performance slice that waits for iteration 22 to MEASURE it first | | Method dispatch · path params · 404 · 405+`Allow` | ✅ | | Wildcards | ⬜ only `:param` today; `*rest` capture is a candidate slice | | Precedence rules | 🔶 registration order IS the rule (documented); specificity-based precedence unneeded until wildcards exist | diff --git a/docs/plan/exploration/fibers/00-fibers.md b/docs/plan/exploration/fibers/00-fibers.md new file mode 100644 index 0000000..f1265d1 --- /dev/null +++ b/docs/plan/exploration/fibers/00-fibers.md @@ -0,0 +1,159 @@ +# Fibers — the concurrency model, its terminology, and why writeonce chose what it chose + +> Exploration/reference note (no status banner by board convention). +> The normative decisions live in the arc spec +> ([`2026-08-20-shard-fiber-arc-design.md`](../../../superpowers/specs/2026-08-20-shard-fiber-arc-design.md)) +> and iterations [8](../../../stories/language-runtime-database/refine/08-shard-actor-runtime.md) / +> [11](../../../stories/language-runtime-database/refine/11-fibers.md); this +> page explains the WHY at doctrine depth. Written 2026-08-20, when this +> file was also the target of a dangling reference from iteration 11 — +> it exists now. + +> **Demonstrated live** by [`docs/examples/fibers`](../../../examples/fibers/README.md) +> (`just fibers`, 8 checks): byte-exact budget interleave, a parked +> sleeper blocking nobody, on the io_uring AND epoll backends. + +## What a green thread is, and why anyone bothers + +A green thread (fiber, virtual thread, coroutine — the tradition picks +the name) is a thread scheduled by the LANGUAGE RUNTIME, not the kernel. +The kernel sees a handful of OS threads; the runtime multiplexes +thousands-to-millions of logical threads on top. + +The economics: an OS thread costs a fixed stack reservation (typically +1–8 MiB of address space) and every context switch goes through the +kernel scheduler (CFS is O(log N), and task counts in the hundreds +already hurt). A green thread can start around 2 KiB and switch in tens +of nanoseconds — a register save and a pointer swap, no syscall. That is +the difference between a few thousand concurrent connections and a few +million. + +## The terminology, untangled + +- **Green thread** — originally Java pre-1.3's model, which was M:1 + (everything on ONE OS thread — no parallelism at all). Now used + loosely for any user-space thread. +- **Fiber** — usually implies COOPERATIVE: it runs until it explicitly + yields. +- **Coroutine** — the broadest term; covers stackful and stackless both. +- **Goroutine** — Go's M:N implementation, preemptively scheduled. +- **Virtual thread** — Java 21's, also M:N. + +The load-bearing distinction between "fiber" and "goroutine" is NOT the +stack — it is the EVICTION POLICY. A classic fiber owns the CPU until it +yields; one selfish loop starves everything behind it. A goroutine can +be interrupted WITHOUT ITS CONSENT (since Go 1.14, via a signal that +stops it at a safe instruction). + +M:1 vs M:N matters concretely here: one process embedding its own +storage would leave every core but one idle under M:1. writeonce is M:N +— stage 2 of the arc pins one kernel thread per core (shards), and +fibers multiplex above them. + +## writeonce's model: cooperative mechanics, accounting-based eviction + +The BEAM shape. Mechanically cooperative — no signals, no interrupts, +no async preemption machinery — but every loop back-edge decrements a +REDUCTION BUDGET (`WO_REDUCTIONS`, default 4000), and at zero the fiber +is re-queued whether it likes it or not. + +What that buys, all three at once: + +- goroutine-grade starvation-freedom (the corpus pins budget-1 as EXACT + round-robin across fibers — `runtime/test/test_fiber.c`); +- fiber-grade DETERMINISM: same program, same schedule, byte-identical + output — signal-based preemption points are nondeterministic by + nature, Go's included; +- zero interruption machinery: an interpreter's dispatch loop is always + at a safe point, so the "where can I safely stop this thread" + problem Go solves with signal handlers and pc maps costs nothing. + +One implementation lesson worth keeping (it cost a livelock): the +budget check runs at loop back-edges AFTER the jump lands, so the saved +resume pc is the loop head. A pre-instruction save at budget 1 +re-executes the jump straight into the same decrement, forever. + +## Should the grammar have `async`? No — permanently rejected + +`async` is only NECESSARY in stackless designs, where the compiler must +know statically which functions can suspend so it can transform each +into a state-machine struct. That requirement is function COLORING: an +async function can only be called from another async function, so the +keyword propagates virally up every call chain, and the ecosystem forks +into sync and async flavors of everything. In writeonce it would color +the query API, then every handler that touches data — the exact code +that most wants to read as straight-line logic. + +With stackful fibers, a blocking read ten frames deep just parks. The +suspension point is a RUNTIME fact, not a TYPE-SYSTEM fact. Same source +text in program mode and server mode; no keyword; no coloring. This is +iteration 11's "async/await — permanently rejected surface, not +deferred", with the reasoning attached. + +## Stackful vs stackless: the real fork, and the recommendation + +**Stackful** (goroutines, Java virtual threads, classic fibers): each +logical thread owns a real, growable stack; it can suspend from ANY +depth. Costs: growth-by-copy must find and rewrite every pointer into +the old stack (Go's machinery), and FFI is the classic killer — C +callees assume a large contiguous stack and know nothing about growth, +so every foreign call needs a stack switch or a guaranteed segment. + +**Stackless** (Rust futures, JS promises, C# async, Python coroutines): +the compiler rewrites each async function into a fixed-size state +object. Cheaper, runtime-optional — and it buys the coloring above, +plus suspension only at explicit await points. + +**Recommendation, and what is already built: STACKFUL — because both of +its classic costs vanish in this stack.** + +1. **No FFI, by doctrine.** The main argument against small growable + stacks is C interop. writeonce has no C to call (the reject row is + load-bearing). The cost is simply absent. +2. **The "stack" is register-indexed, not address-based.** A + `wo_fiber`'s state is the interpreter's register window + frame + array — frames reference registers by INDEX, so a context is + relocatable by construction. Growth is allocate-larger + memcpy with + ZERO pointer fixups; Go's hardest stackful problem does not exist + here. Contexts are ~42 KiB fixed today; start-small (~4 KiB) + growable contexts are the recorded improvement that takes fiber + counts from thousands toward millions (arc spec, "Fiber context + growth"). + +## The two problems every design must solve, and where they land here + +- **Blocking syscalls.** A green thread calling fsync blocks the OS + thread carrying it and strands every fiber behind it — and an + embedded database fsyncs on every commit, milliseconds each, an + eternity for a scheduler. writeonce's answer is io_uring, twice: the + arc's per-shard ring parks blocking `net`/`time` builtins (T4, + readiness ops first, then reads/writes as ring ops), and iteration + 23 rides the SAME ring for WAL WRITE+FSYNC group-commit so the + DB-owner shard keeps executing while durability drains. epoll + survives only as the portability fallback behind a startup probe + (`WO_IO=uring|epoll`) — seccomp'd containers routinely deny io_uring, + and the binary must run everywhere. The reference cards live in + [`../linux/07-io_uring.md`](../linux/07-io_uring.md). +- **Preemption.** Solved by the reduction budget above. And to the + standing question "does preemption depend on FFI?" — NO, inversely: + preemption holes come from frames the runtime cannot interrupt, which + in Go means foreign C frames (its signal preemption skips them). No + FFI means no foreign frames; the only non-preemptible regions are our + OWN builtins — C we control, bounded, and the blocking ones become + parked ring ops. FFI would have CREATED the dependency; its absence + is why the budget is airtight. + +## Precedent survey (what was adopted, what was rejected) + +- **BEAM (Erlang/Elixir): adopted** — reduction-budget preemption, + actor mailboxes, one-message-at-a-time delivery, isolated failure + (a fiber's uncaught trap kills that fiber alone). +- **Go: half-adopted** — M:N over pinned threads yes; stack-copy + pointer rewriting unnecessary here (register-indexed contexts); + signal-based async preemption rejected (nondeterministic, and the + budget makes it redundant). +- **Rust/Tokio, JS, C#, Python: rejected** — stackless coloring is the + cost this language exists to not pay. +- **Java Loom: matches** — park-under-a-blocking-API is exactly the + stdlib posture (`net.read` blocks in program mode, parks under + fibers, same signature). diff --git a/docs/stories/language-runtime-database/00-story.md b/docs/stories/language-runtime-database/00-story.md index fce25a2..bd53522 100644 --- a/docs/stories/language-runtime-database/00-story.md +++ b/docs/stories/language-runtime-database/00-story.md @@ -40,7 +40,12 @@ iterations); no commits by agents — drafts go to `.dev/commit.md`. ## Iterations (rows in dependency order — `#` is an immutable ID, not a rank) -Re-sequenced 2026-08-20 from the edges in +RENUMBERED 2026-08-20 (developer directive): pending iterations carry +fresh IDs in priority order; LANDED iterations keep their historical +numbers (code comments and commit history cite them — records, not a +queue). Mapping: 19←20(scalars), 20←9c, 21←9d, 22←9e, 23←9f, 24←19(chat), +25←10, 26←12, 27←9g, 28←14, 29←13; 8, 11, 17, 18 unchanged. +Re-sequenced from the edges in [`docs/00-dependency-graph.md`](../../00-dependency-graph.md): landed rows in landing order, then the pending rows in implementation order. File names keep their IDs — every board, spec, and plan references @@ -52,28 +57,29 @@ iterations by number, so numbers never renumber. | 2 | 2 | [VM core](done/02-vm-core.md) | `wovm`: `.wob` loader, register interpreter, arena, borrow word, `@gc` collector | | 3 | 3 | [Compiler front](done/03-compiler-front.md) | `woc`: lexer → parser → typechecker → ownership pass, diagnostics | | 4 | 4 | [Single binary end-to-end](done/04-single-binary-e2e.md) | emitter + conformance corpus + `woc build` self-contained binary | -| 5 | 5 | [Language surface](05-language-surface.md) | Haxe-parity adoptions: switch, records, optionals, try/catch, statics, modules… (`pub(read)`/`using`/`#if` remainders sit in the post-12 drain) | +| 5 | 5 | [Language surface](05-language-surface.md) | Haxe-parity adoptions, grammar + strictness halves (landed in waves through 2026-08-20) | | 6 | 6 | [Program mode + stdlib](done/06-program-mode-stdlib.md) | `fn main`, exit codes, `fs`/`proc`/`net`/`time`/`json` builtins | | 7 | 7 | [log-watcher proof](done/07-logwatcher-proof.md) | the driving workload compiled, executable, soak-proven (landed 2026-08-15) | -| 8 | 7b | [Inferred GC + mark-sweep](done/07b-inferred-gc-mark-sweep.md) | `@gc` removed from the language; compiler infers GC-ness; RC replaced by incremental per-shard tri-color mark-sweep | +| 8 | 7b | [Inferred GC + mark-sweep](done/07b-inferred-gc-mark-sweep.md) | `@gc` removed; GC-ness inferred; RC replaced by incremental per-shard tri-color mark-sweep | | 9 | 9 | [Database engine](done/09-database-engine.md) | class-shaped tables, typed WAL + recovery, `insert`/`select` execute | -| 10 | 9b | [`@table`, relations, query](done/09b-table-relations-query.md) | `@table` becomes real storage; typed `ref`/`backlink`/`multi` relations; compiler-checked LINQ-shaped queries lowered to engine ops | -| 11 | 15 | [deps: `wo.toml [deps]`](done/15-deps-package-manager.md) | exact-rev git dependencies + `wo.lock` + `.wo-deps` cache; `use ` resolves a fetched project as a module root; flat-only, network-free when locked | -| 12 | 16 | [web framework](done/16-web-framework.md) | a `.wo`-library framework (HTTP/1.1 keep-alive behind a TLS-terminating proxy): router, `Handler`/`Middleware` interfaces, auth, all three body hooks, `@table` data layer; `docs/examples/web-app` consumes it via `[deps]`; h2c parked behind 8/9f/11 | -| **13** | **18** | [framework v2: memory-rich features](18-memory-db-features.md) | **NEXT — spec approved 2026-08-20**: TTL cache, @table feature flags with cached reads, durable @table job queue with drain-on-request, `transaction { }` exposing the WAL's staged batch (enqueue + write, one commit — no outbox) | -| 14 | 9c | [Cross-program tables](refine/09c-cross-program-tables.md) | attach to a running program's database over a local IPC channel: manifest-granted rights, typed statements, owner stays the single writer (channel half-built) | -| 15 | 9d | [Keypair attach auth](refine/09d-keypair-attach-auth.md) | program identity is a keypair: mutual challenge–response at attach, grants name public keys, replay-proof (crypto half-built; plan folds into 9c's) | -| 16 | 9e | [Durability, throughput, scale](refine/09e-durability-throughput-scale.md) | restart-persistence proof, read/write benchmark, ~1M-row load; the baseline 8/9f/11 sign against | -| 17 | 8 | [Shard-actor runtime](08-shard-actor-runtime.md) | thread-per-core shards, per-shard heaps, ownership-move messaging (collector precondition met by 7b) | -| 18 | 9f | [io_uring group-commit](refine/09f-io-uring-commit.md) | replace fsync-per-commit with io_uring batched durability on the shard tick; fsync fallback kept (after 8 + 9e, explicit) | -| 19 | 11 | [Fibers](refine/11-fibers.md) | green threads on the shard scheduler: reduction-budget preemption, blocking builtins park; unparks h2c (with 8/9f), streaming, cancellation, pub/sub, fiber jobs | -| 20 | 10 | [HTTP service layer](10-http-service.md) | `service` blocks lower onto the framework (after 9b + 9c by their own precedence notes) | -| 21 | 12 | [Blue-green deploy](12-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback (plan authored after 9 + 10) | -| 22 | 9g | [Query grammar corpus](refine/09g-query-grammar-corpus.md) | grow the query grammar from real corpora; likely collapses to "confirm `len(query)` + add `exists`"; precedes 14 | -| 23 | 14 | [skillhost host workload](refine/14-skillhost-host-workload.md) | host-shaped driving workload naming runtime gaps (bounded subprocess, stdin/stdout transport, fs metadata, FFI-vs-out-of-process) — demoted with the framework goal | -| 24 | 13 | [Compile-time metaprogramming](refine/13-compile-time-metaprogramming.md) | `@derive(Json/Csv/Eq/Hash/Show)` from class-table metadata; held behind 12 with the parked drain by the 2026-08-08 scope directive | -| 26 | 20 | [Float + Bytes](20-missing-scalar-types.md) | the missing scalars, full stack: IEEE-quiet f64 through literals/VM/@table/WAL/json (fractions decode at last) + Bytes as the binary carrier; forks settled 2026-08-20, spec next — feeds 19 (WS frames) and the crypto fork (digests) | -| ⏸ | 17 | [library projects + `internal/`](17-library-projects-internal.md) | **PARKED** (spec + plan approved, branch `library-internal`) — `wo.toml` kind = "library" + Go's `internal/` rule; slots anywhere after 16 whenever directed, bringing the framework reorg with it | +| 10 | 9b | [`@table`, relations, query](done/09b-table-relations-query.md) | `@table` real storage; `ref`/`backlink`/`multi`; compiler-checked queries | +| 11 | 15 | [deps: `wo.toml [deps]`](done/15-deps-package-manager.md) | exact-rev git deps + `wo.lock` + `.wo-deps`; flat-only, offline once locked | +| 12 | 16 | [web framework](done/16-web-framework.md) | the `.wo` framework v1 (router, middleware, auth, all three body hooks) consumed via `[deps]` | +| 13 | 8+11 | [Shard-actor runtime](08-shard-actor-runtime.md) · [Fibers](refine/11-fibers.md) | THE ARC (stages 1+2 landed 2026-08-20: fibers/budget/actors/io_uring plane; pinned shards, envelope sends, home-routed frees, WO-E222); stage 3 = transparent DB RPC + 22 re-run | +| 14 | 18 | [framework v2: memory-rich features](18-memory-db-features.md) | spec+plan approved: TTL cache, @table flags, durable job queue, `transaction { }` over the WAL's staged batch | +| 15 | 19 | [Float + Bytes](19-missing-scalar-types.md) | the missing scalars, full stack: IEEE-quiet f64 through literals/VM/@table/WAL/json + Bytes as the binary carrier — feeds 24 (WS frames) and the crypto fork (digests). *(was 20)* | +| 16 | 20 | [Cross-program tables](refine/20-cross-program-tables.md) | attach to a running program's database over local IPC; owner stays the single writer (channel half-built). *(was 9c)* | +| 17 | 21 | [Keypair attach auth](refine/21-keypair-attach-auth.md) | program identity is a keypair; mutual challenge–response at attach (crypto half-built; plan folds into 20's). *(was 9d)* | +| 18 | 22 | [Durability, throughput, scale](refine/22-durability-throughput-scale.md) | restart-persistence proof, benchmarks, ~1M rows — the baseline the arc and 23 sign against. *(was 9e)* | +| 19 | 23 | [io_uring group-commit](refine/23-io-uring-commit.md) | WAL WRITE+FSYNC chains on the arc's per-shard rings; fsync fallback kept (after 22 + the arc). *(was 9f)* | +| 20 | 24 | [chat: WebSocket workload](refine/24-chat-websocket-workload.md) | the arc's acceptance: WS upgrade + frames (SHA-1 via crypto fork, Bytes via 19), rooms/broadcast, 1k clients, drain-clean. *(was 19)* | +| 21 | 25 | [HTTP service layer](25-http-service.md) | `service` blocks lower onto the framework (after 9b + 20 by their own precedence notes). *(was 10)* | +| 22 | 26 | [Blue-green deploy](26-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback (plan authored after 9 + 25). *(was 12)* | +| 23 | 27 | [Query grammar corpus](refine/27-query-grammar-corpus.md) | grow the query grammar from real corpora; likely collapses to "confirm `len(query)` + add `exists`"; precedes 28. *(was 9g)* | +| 24 | 28 | [skillhost host workload](refine/28-skillhost-host-workload.md) | host-shaped driving workload naming runtime gaps — demoted with the framework goal. *(was 14)* | +| 25 | 29 | [Compile-time metaprogramming](refine/29-compile-time-metaprogramming.md) | `@derive(...)` from class-table metadata; held with the parked drain by the 2026-08-08 scope directive. *(was 13)* | +| ⏸ | 17 | [library projects + `internal/`](17-library-projects-internal.md) | **PARKED** (spec + plan approved, branch `library-internal`) — `wo.toml` kind = "library" + Go's `internal/` rule; slots anywhere after 16 on directive | + Review protocol: the developer reads one iteration, approves or amends; the next starts only after approval. Each iteration is an unsplittable @@ -104,8 +110,8 @@ list, and a pointer to the plan document that already sequences its tasks. **parked** with spec + plan ready on branch `library-internal`. The v1 slices landed 2026-08-20 (`just web-app` 21/0 after polish, auth, form, multipart). Implementation order for everything still pending: - **18 (spec approved)** → 9c/9d → 9e → 8 → 9f → 11 (+ h2c unparks) → - 10 → 12 → 9g → 14 → 13 + parked drain; 17 parked, slots anywhere after + **18 (spec approved)** → 20/21 → 22 → 8 → 23 → 11 (+ h2c unparks) → + 10 → 12 → 27 → 14 → 13 + parked drain; 17 parked, slots anywhere after 16 on directive. The iterations table above carries this order row-by-row; edges live in [`docs/00-dependency-graph.md`](../../00-dependency-graph.md). diff --git a/docs/stories/language-runtime-database/08-shard-actor-runtime.md b/docs/stories/language-runtime-database/08-shard-actor-runtime.md index 15efcb9..6afd1fc 100644 --- a/docs/stories/language-runtime-database/08-shard-actor-runtime.md +++ b/docs/stories/language-runtime-database/08-shard-actor-runtime.md @@ -45,9 +45,9 @@ - **Gated by the benchmark (2026-08-15):** this is the "optimize multithreading" lever of the performance arc — thread-per-core is a throughput/scale claim, so landing it means re-running iteration - [9e](refine/09e-durability-throughput-scale.md) at the connection/concurrency + [22](refine/22-durability-throughput-scale.md) at the connection/concurrency scale it unlocks and recording the before/after delta. It is also where - the io_uring write path ([9f](refine/09f-io-uring-commit.md)) gets a thread to + the io_uring write path ([23](refine/23-io-uring-commit.md)) gets a thread to overlap durability against. ## Proposed Solution diff --git a/docs/stories/language-runtime-database/18-memory-db-features.md b/docs/stories/language-runtime-database/18-memory-db-features.md index 48f3110..efb1adc 100644 --- a/docs/stories/language-runtime-database/18-memory-db-features.md +++ b/docs/stories/language-runtime-database/18-memory-db-features.md @@ -53,8 +53,8 @@ worked around. writes. Honest limit stated everywhere it matters: an idle server drains nothing until the next request arrives. Fibers (11) later replaces the scheduler; the queue table and job shape stay. - (Rejected for v1: a second worker process over 9c attach — real - parallelism but blocks on finishing 9c; parking jobs entirely — the + (Rejected for v1: a second worker process over 20 attach — real + parallelism but blocks on finishing 20; parking jobs entirely — the queue-plus-drain is useful today.) 3. **`transaction { }` ships in this iteration.** Language block deferring `wal_commit` to block end; a trap unwinding out of the block aborts the diff --git a/docs/stories/language-runtime-database/20-missing-scalar-types.md b/docs/stories/language-runtime-database/19-missing-scalar-types.md similarity index 96% rename from docs/stories/language-runtime-database/20-missing-scalar-types.md rename to docs/stories/language-runtime-database/19-missing-scalar-types.md index d5ae3d9..448518d 100644 --- a/docs/stories/language-runtime-database/20-missing-scalar-types.md +++ b/docs/stories/language-runtime-database/19-missing-scalar-types.md @@ -1,4 +1,4 @@ -# Iteration 20 — the missing scalar types: Float and Bytes +# Iteration 19 — the missing scalar types: Float and Bytes > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](00-story.md). @@ -30,9 +30,9 @@ work, but it blurs every json/interpolation boundary Text has. and `trunc(f)` are the explicit bridges. 3. **Bytes ships alongside**: a distinct binary scalar (len/byte_at/ slice/compare; base64 and future digests return it; net reads can - fill it) — Text goes back to meaning text; iteration 19 and the + fill it) — Text goes back to meaning text; iteration 24 and the crypto fork inherit a clean carrier. -4. **Recorded as iteration 20; spec before code.** +4. **Recorded as iteration 19; spec before code.** ## Surveyed and deliberately NOT added (the rest of the missing-type list) diff --git a/docs/stories/language-runtime-database/10-http-service.md b/docs/stories/language-runtime-database/25-http-service.md similarity index 98% rename from docs/stories/language-runtime-database/10-http-service.md rename to docs/stories/language-runtime-database/25-http-service.md index 9a2a2b7..625daf5 100644 --- a/docs/stories/language-runtime-database/10-http-service.md +++ b/docs/stories/language-runtime-database/25-http-service.md @@ -1,4 +1,4 @@ -# Iteration 10 — HTTP service layer +# Iteration 25 — HTTP service layer > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](00-story.md). diff --git a/docs/stories/language-runtime-database/12-blue-green-deploy.md b/docs/stories/language-runtime-database/26-blue-green-deploy.md similarity index 98% rename from docs/stories/language-runtime-database/12-blue-green-deploy.md rename to docs/stories/language-runtime-database/26-blue-green-deploy.md index 968ce1d..a28c751 100644 --- a/docs/stories/language-runtime-database/12-blue-green-deploy.md +++ b/docs/stories/language-runtime-database/26-blue-green-deploy.md @@ -1,4 +1,4 @@ -# Iteration 12 — blue-green in-runtime deployment +# Iteration 26 — blue-green in-runtime deployment > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](00-story.md). diff --git a/docs/stories/language-runtime-database/done/07b-inferred-gc-mark-sweep.md b/docs/stories/language-runtime-database/done/07b-inferred-gc-mark-sweep.md index f2e0525..5ea8ec0 100644 --- a/docs/stories/language-runtime-database/done/07b-inferred-gc-mark-sweep.md +++ b/docs/stories/language-runtime-database/done/07b-inferred-gc-mark-sweep.md @@ -92,7 +92,7 @@ - **Gated by the benchmark (2026-08-15):** this is the "implement garbage collection" lever of the performance arc — tri-color mark-sweep replacing RC changes the write path's tail latency, so landing it means re-running - iteration [9e](../refine/09e-durability-throughput-scale.md) and recording the + iteration [22](../refine/22-durability-throughput-scale.md) and recording the delta (does tracing help or hurt p99 under write load?). - **Constraint added by the database track (2026-08-15):** a GC-managed value in a `@table` field is a compile error (the engine/heap bulkhead — 9b diff --git a/docs/stories/language-runtime-database/done/09b-table-relations-query.md b/docs/stories/language-runtime-database/done/09b-table-relations-query.md index ad196e4..1da4ac4 100644 --- a/docs/stories/language-runtime-database/done/09b-table-relations-query.md +++ b/docs/stories/language-runtime-database/done/09b-table-relations-query.md @@ -5,7 +5,7 @@ > > **Inserted 2026-08-11**, hence `9b` rather than a renumber. It follows > iteration 9 because a query surface needs tables that actually execute, and -> precedes iteration 10 because `service` blocks will want to return query +> precedes iteration 25 because `service` blocks will want to return query > results. > > **Spec exists (2026-08-15):** @@ -74,7 +74,7 @@ HTTP/UI track's; a query that pushes updates is a later composition of the two. - Migrations. Changing a `@table` class's shape is the blue-green spec's - additive-only differ (iteration 12), not this iteration's problem. + additive-only differ (iteration 26), not this iteration's problem. - Query optimisation beyond index selection. A cost-based planner is a separate, much later concern; this iteration must only prove that declared indexes are used. diff --git a/docs/stories/language-runtime-database/done/16-web-framework.md b/docs/stories/language-runtime-database/done/16-web-framework.md index d417330..154419f 100644 --- a/docs/stories/language-runtime-database/done/16-web-framework.md +++ b/docs/stories/language-runtime-database/done/16-web-framework.md @@ -58,7 +58,7 @@ > like any dependency (iteration 15 is the prerequisite). TLS terminates at a > reverse proxy — browsers get TLS+ALPN+h2 from nginx/caddy while the > framework speaks HTTP/1.1 keep-alive behind it, so no TLS exists anywhere -> in the toolchain. h2c is the parked successor (after iterations 8/9f/11, +> in the toolchain. h2c is the parked successor (after iterations 8/23/11, > when multiplexing has a scheduler to pay off on). > > **Spec exists:** [`2026-08-18-web-framework-design.md`](../../../superpowers/specs/2026-08-18-web-framework-design.md) @@ -88,7 +88,7 @@ (concurrency arrives underneath via iterations 8/11), no chunked encoding, no WebSockets, JSON-first (no templates — the removed UI track stays removed). -- **Relationship to iteration 10 recorded in both**: `service` blocks later +- **Relationship to iteration 25 recorded in both**: `service` blocks later *lower onto this library* — compiler sugar over the same router, never a rival stack. @@ -111,9 +111,9 @@ ## Out Of Scope TLS in the toolchain (proxy-terminated by decision); HTTP/2 + the -bytes/buffer type (parked to the h2c successor, after 8/9f/11); chunked +bytes/buffer type (parked to the h2c successor, after 8/23/11); chunked transfer encoding; WebSockets/SSE; templates/SSR; multipart uploads; -performance work beyond the soak's flatness gate (benchmarks belong to 9e's +performance work beyond the soak's flatness gate (benchmarks belong to 22's measurement backbone). ## Proposed Solution diff --git a/docs/stories/language-runtime-database/refine/11-fibers.md b/docs/stories/language-runtime-database/refine/11-fibers.md index 5c0e9ff..85d2d6c 100644 --- a/docs/stories/language-runtime-database/refine/11-fibers.md +++ b/docs/stories/language-runtime-database/refine/11-fibers.md @@ -39,7 +39,7 @@ - **Given** a parked fiber at shard shutdown, - **when** the shard unwinds it, - **then** every drop map runs (ASan zero leaks) — parked fibers die - as cleanly as trapped ones. (Iteration 12's blue-green drain reuses + as cleanly as trapped ones. (Iteration 26's blue-green drain reuses exactly this unwind path.) - What to achieve? - **Given** `@gc` objects referenced only from a parked fiber's frames, diff --git a/docs/stories/language-runtime-database/refine/09c-cross-program-tables.md b/docs/stories/language-runtime-database/refine/20-cross-program-tables.md similarity index 95% rename from docs/stories/language-runtime-database/refine/09c-cross-program-tables.md rename to docs/stories/language-runtime-database/refine/20-cross-program-tables.md index 59aea78..cd76390 100644 --- a/docs/stories/language-runtime-database/refine/09c-cross-program-tables.md +++ b/docs/stories/language-runtime-database/refine/20-cross-program-tables.md @@ -1,12 +1,12 @@ -# Iteration 9c — cross-program tables: attach to a running program's database +# Iteration 20 — cross-program tables: attach to a running program's database > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](../00-story.md). > -> **Inserted 2026-08-15**, hence `9c`. It follows 9b because a program +> **Inserted 2026-08-15**, hence `20`. It follows 9b because a program > attaching to another's tables wants the same typed statements and queries > the owner has — a surface that must exist before it can be shared — and -> precedes iteration 10 because HTTP is the *external* face of a program; +> precedes iteration 25 because HTTP is the *external* face of a program; > this iteration is the *writeonce-native* face, program to program on the > same machine. > @@ -72,7 +72,7 @@ ## Out Of Scope - **Remote machines.** The IPC string names a local channel; cross-host - access is the HTTP/service layer's job (iteration 10) or a much later + access is the HTTP/service layer's job (iteration 25) or a much later network protocol. Same-machine is what "attach" means here. - **B caching A's rows.** Every read crosses the channel; a client-side cache (and its invalidation) is a later performance iteration, if ever. @@ -84,7 +84,7 @@ subscription registry later (the client-api phase doc already sketches the wire shape). - **Schema migration while attached** — a blue-green swap in A while B - holds an attachment is iteration 12's compatibility problem; this + holds an attachment is iteration 26's compatibility problem; this iteration may simply drop attachments on swap. ## Info @@ -136,8 +136,8 @@ read or read+write (per-table refinement deferred until a workload needs it), and the registration is A's manifest so a grant is a config change + restart, not an API. **Superseded as the end state (2026-08-15):** identity is a keypair and grants name public keys — iteration -[9d](09d-keypair-attach-auth.md) owns that; the uid check is only this -iteration's bootstrap and must be flagged pre-9d wherever it ships. +[21](21-keypair-attach-auth.md) owns that; the uid check is only this +iteration's bootstrap and must be flagged pre-21 wherever it ships. **4. What does B's statement actually block on?** B's insert crosses the channel, executes in A (RAM + WAL + fsync), and acknowledges back — a diff --git a/docs/stories/language-runtime-database/refine/09d-keypair-attach-auth.md b/docs/stories/language-runtime-database/refine/21-keypair-attach-auth.md similarity index 91% rename from docs/stories/language-runtime-database/refine/09d-keypair-attach-auth.md rename to docs/stories/language-runtime-database/refine/21-keypair-attach-auth.md index b2e3c8f..39c74eb 100644 --- a/docs/stories/language-runtime-database/refine/09d-keypair-attach-auth.md +++ b/docs/stories/language-runtime-database/refine/21-keypair-attach-auth.md @@ -1,13 +1,13 @@ -# Iteration 9d — keypair authentication for cross-program attach +# Iteration 21 — keypair authentication for cross-program attach > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](../00-story.md). > -> **Inserted 2026-08-15.** Promotes iteration 9c's identity fork (Info, +> **Inserted 2026-08-15.** Promotes iteration 20's identity fork (Info, > fork 3) to its own iteration: the name + unix-uid lean is the milestone > bootstrap, and THIS is what replaces it — program identity is a keypair, > and an attachment is granted to a public key, not to a process that -> happens to share a uid. It follows 9c (there is nothing to authenticate +> happens to share a uid. It follows 20 (there is nothing to authenticate > until attach exists) and stays same-machine; the same handshake is what > a future remote channel would reuse, which is the point of doing it > properly now. @@ -21,7 +21,7 @@ manifest) and a public key it can print/export. Identity stops being "whoever reached the socket first with the right uid". - **Grants name public keys.** A's `[share]` registers a client by its - public key (fingerprint), with rights exactly as 9c defined them; B's + public key (fingerprint), with rights exactly as 20 defined them; B's `[connect.a]` **pins A's public key** beside the IPC string. Both sides authenticate: A proves it is A before B sends a byte of intent, B proves it is B before A executes a statement. @@ -38,7 +38,7 @@ read+write, and B's `[connect.a]` pinning A's public key, - **when** B attaches, - **then** the mutual handshake completes, the attachment carries B's - granted rights, and every 9c acceptance behavior (statements, traps, + granted rights, and every 20 acceptance behavior (statements, traps, refusals) holds unchanged on top of it. - What to achieve? - **Given** a client presenting a keypair A never registered, @@ -47,7 +47,7 @@ client sees the catchable authentication trap, and A logs the offered fingerprint (so granting it is a copy-paste, not an investigation). - What to achieve? - - **Given** a same-uid process (the 9c bootstrap's whole trust basis) + - **Given** a same-uid process (the 20 bootstrap's whole trust basis) presenting no key or the wrong key, - **when** it attempts to attach, - **then** it is refused — proving the uid check has been superseded, @@ -128,11 +128,11 @@ authorization input. ## Proposed Solution - **Brainstorm the spec** settling the four forks, then fold the plan into - 9c's implementation plan as its authentication tasks — one plan, because - 9c without 9d ships a placeholder identity and 9d without 9c has nothing - to authenticate. The 9c milestone may still land first with the uid - bootstrap, flagged loudly as pre-9d. -- **Acceptance extends the 9c workload**: the employee-A / + 20's implementation plan as its authentication tasks — one plan, because + 20 without 21 ships a placeholder identity and 21 without 20 has nothing + to authenticate. The 20 milestone may still land first with the uid + bootstrap, flagged loudly as pre-21. +- **Acceptance extends the 20 workload**: the employee-A / employee-list-B pair (`docs/examples/employee-list`, pre-authored 2026-08-15) carries the key exchange in both manifests — A's `[[share.clients]]` names B's fingerprint, B's `[connect.employee]` pins diff --git a/docs/stories/language-runtime-database/refine/09e-durability-throughput-scale.md b/docs/stories/language-runtime-database/refine/22-durability-throughput-scale.md similarity index 91% rename from docs/stories/language-runtime-database/refine/09e-durability-throughput-scale.md rename to docs/stories/language-runtime-database/refine/22-durability-throughput-scale.md index 400049b..3c433e2 100644 --- a/docs/stories/language-runtime-database/refine/09e-durability-throughput-scale.md +++ b/docs/stories/language-runtime-database/refine/22-durability-throughput-scale.md @@ -1,4 +1,4 @@ -# Iteration 9e — durability proof, throughput, and scale under load +# Iteration 22 — durability proof, throughput, and scale under load > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](../00-story.md). @@ -6,7 +6,7 @@ > **Inserted 2026-08-15.** The measurement backbone. Everything after the > functional engine (9/9b) is an *optimization*, and an optimization without > a number is a guess — this iteration is the number. It comes before the -> optimization iterations (7b GC, 8 shard-actor, 9f io_uring) reopen for +> optimization iterations (7b GC, 8 shard-actor, 23 io_uring) reopen for > performance work, because each of those must be gated by re-running THIS > iteration's benchmark and showing the number moved the right way. > @@ -29,7 +29,7 @@ a stated duration, with throughput and tail latency inside a stated budget and RSS flat (the log-watcher soak discipline, at database scale). - **The benchmark is the contract every later optimization signs.** 7b (GC), - 8 (shard-actor threads), and 9f (io_uring) each re-run this and record the + 8 (shard-actor threads), and 23 (io_uring) each re-run this and record the before/after — no optimization lands without a measured delta. ## Acceptance Criteria @@ -58,14 +58,14 @@ a durable configuration — a kill mid-load followed by replay loses no acknowledged write. - What to achieve? - - **Given** any later optimization iteration (7b, 8, 9f), + - **Given** any later optimization iteration (7b, 8, 23), - **when** it claims a speedup, - **then** this benchmark's before/after numbers are in that iteration's record, and a claim with no measured delta is not accepted. ## Out Of Scope -- **The optimizations themselves.** This iteration MEASURES; 7b/8/9f change. +- **The optimizations themselves.** This iteration MEASURES; 7b/8/23 change. A single-thread RAM-authoritative baseline is a legitimate first number — the point is to have one before anyone tunes. - **Distributed / multi-machine load.** Same-machine, one process (or one @@ -114,7 +114,7 @@ a bounded concurrent read/write workload here; connection scale deferred to the honest number for a durable workload; RAM-only (no `WO_DATA`) measures the engine's ceiling. Both matter and mean different things. Leaning: publish both, labeled — durable is the number an operator plans against, and -the gap between them is precisely what iteration 9f (io_uring group-commit) +the gap between them is precisely what iteration 23 (io_uring group-commit) exists to close. ## Proposed Solution @@ -123,13 +123,13 @@ exists to close. task is the harness and the baseline file, because nothing downstream means anything without them. - **Sequence the whole performance arc around this iteration:** - 1. 9b lands → employee compiles and runs → **9e restart-persistence** and - **9e baseline benchmark** (single-thread, both durable and RAM-only). - 2. **7b** (inferred GC + mark-sweep) → re-run 9e, record the delta (does + 1. 9b lands → employee compiles and runs → **22 restart-persistence** and + **22 baseline benchmark** (single-thread, both durable and RAM-only). + 2. **7b** (inferred GC + mark-sweep) → re-run 22, record the delta (does tracing change the write path's tail latency?). - 3. **8** (shard-actor, thread-per-core) → re-run 9e at the connection/ + 3. **8** (shard-actor, thread-per-core) → re-run 22 at the connection/ concurrency scale it unlocks, record the delta. - 4. **9f** (io_uring group-commit) → re-run 9e's durable write number, record + 4. **23** (io_uring group-commit) → re-run 22's durable write number, record the delta against the fsync-per-commit baseline — the payoff. - The benchmark harness and its baseline live under `bench/` (or the existing `runtime/bench/`), and `just` gets a `db-bench` recipe kept off the fast diff --git a/docs/stories/language-runtime-database/refine/09f-io-uring-commit.md b/docs/stories/language-runtime-database/refine/23-io-uring-commit.md similarity index 90% rename from docs/stories/language-runtime-database/refine/09f-io-uring-commit.md rename to docs/stories/language-runtime-database/refine/23-io-uring-commit.md index 64e9822..fb43e60 100644 --- a/docs/stories/language-runtime-database/refine/09f-io-uring-commit.md +++ b/docs/stories/language-runtime-database/refine/23-io-uring-commit.md @@ -1,11 +1,11 @@ -# Iteration 9f — io_uring group-commit write path +# Iteration 23 — io_uring group-commit write path > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](../00-story.md). > > **Inserted 2026-08-15.** The write-path optimization, and deliberately the > LAST database performance iteration: it only earns its complexity once -> there is a measured fsync-per-commit baseline to beat (iteration 9e) and a +> there is a measured fsync-per-commit baseline to beat (iteration 22) and a > multithreaded runtime to overlap against (iteration 8). Doing it earlier > would optimize a number nobody had measured, against a runtime that > couldn't use it. @@ -24,21 +24,21 @@ statements while the ring drains, instead of blocking one thread on one fdatasync — the multithreading the throughput number has been waiting for. - **Keep the durability promise byte-for-byte.** Every guarantee iterations 9 - and 9e proved — replay-whole-or-not-at-all, torn-tail drop, no + and 22 proved — replay-whole-or-not-at-all, torn-tail drop, no acknowledged write ever lost — holds identically; io_uring changes HOW the bytes reach the platter, never WHETHER an ack means durable. ## Acceptance Criteria - What to achieve? - - **Given** the io_uring write path under the iteration-9e crash battery + - **Given** the io_uring write path under the iteration-22 crash battery (concurrent writers, kill -9 mid-stream, reboot, replay), - **when** it runs, - **then** every acknowledged write is present after replay and no unacknowledged partial write is ever visible — the exact result the fsync path gives, so durability is provably unchanged. - What to achieve? - - **Given** the iteration-9e durable write benchmark, + - **Given** the iteration-22 durable write benchmark, - **when** it is run on the fsync-per-commit path and then the io_uring group-commit path on the same machine, - **then** the io_uring path's write throughput is materially higher and @@ -61,7 +61,7 @@ read path. This is a write-durability optimization, full stop. - **Registered buffers / fixed files / SQPOLL tuning** beyond what the benchmark shows is worth it. Start with the plain submit/complete model; - add ring features only when 9e's number says a specific one pays. + add ring features only when 22's number says a specific one pays. - **Replacing the WAL format or the commit contract.** The bytes on disk and the meaning of an ack are iteration 9's; this changes the syscall, not the format. @@ -78,7 +78,7 @@ wrap it behind the existing `wo_wal_commit` boundary (drop-in, the engine never learns) or expose an async-commit primitive the shard scheduler drives (faster overlap, but couples the WAL to iteration 8's loop). Leaning: drop-in behind `wo_wal_commit` first — it is the correctness-preserving -step and 9e can measure it standalone — then an async variant only if 8's +step and 22 can measure it standalone — then an async variant only if 8's scheduler shows the blocking boundary is the remaining bottleneck. **2. liburing or raw syscalls?** liburing is the ergonomic wrapper but is a @@ -104,13 +104,13 @@ auto-probe is what production uses. ## Proposed Solution -- **Brainstorm the spec** after iterations 8 and 9e exist — this iteration is +- **Brainstorm the spec** after iterations 8 and 22 exist — this iteration is meaningless without a multithreaded runtime to overlap against and a - measured baseline to beat, and its plan's acceptance is literally "9e's - durable number improved, 9e's crash battery still green, fsync fallback + measured baseline to beat, and its plan's acceptance is literally "22's + durable number improved, 22's crash battery still green, fsync fallback still correct". - Expected shape: a `wo_wal` write-mode switch (fsync vs uring), the raw ring setup + submit/complete in `database/src/wal.c` (or a `wal_uring.c` beside it), the startup probe + `WO_WAL_MODE` override, the binding doc's - WAL section extended with the ring layout, and iteration 9e re-run on both + WAL section extended with the ring layout, and iteration 22 re-run on both paths with the delta committed. diff --git a/docs/stories/language-runtime-database/refine/24-chat-websocket-workload.md b/docs/stories/language-runtime-database/refine/24-chat-websocket-workload.md new file mode 100644 index 0000000..aa0cdbc --- /dev/null +++ b/docs/stories/language-runtime-database/refine/24-chat-websocket-workload.md @@ -0,0 +1,69 @@ +# Iteration 24 — chat: the WebSocket pub/sub driving workload + +> Format: fiberloom `product/story-iteration-template`. Part of +> [Story — one language, one runtime, one database, one binary](../00-story.md). +> +> **Inserted 2026-08-20** (concurrency-chain refinement): the 8+11 arc's +> driving workload, the role log-watcher played for iterations 3–7. Needs +> its spec AFTER the arc's — it lands at the arc's end and proves it. + +## Why this iteration exists + +Everything the framework ledger parks behind concurrency — WebSockets, +pub/sub, streaming, per-request cancellation, the keep-alive parking +retirement — needs a workload that actually exercises long-lived +connections and cross-shard broadcast, or the arc ships mechanism without +proof. Chat is the smallest honest such workload: rooms, N concurrent +clients, fan-out on every message, presence on connect/disconnect — one +binary, no broker. + +## Goals + +- `docs/examples/chat`: rooms + broadcast + presence over WebSocket, + served by the framework through `[deps]` exactly as the web-app is. +- Framework grows the WS mechanism in pure `.wo`: the HTTP/1.1 upgrade + handshake (Sec-WebSocket-Accept needs SHA-1/base64 — the crypto-builtin + fork's first real consumer), frame parse/serialize (text, close, ping), + fiber-per-connection serving (iteration 11), and in-memory channels + whose delivery is cross-shard message send (iteration 8). +- The arc's acceptance teeth: 1k concurrent clients across shards, + broadcast latency measured, starvation-free under one hot room, + SIGTERM drains every connection cleanly. + +## Acceptance Criteria (draft — the spec after the arc refines) + +- **Given** two clients in one room on DIFFERENT shards, **when** one + sends, **then** the other receives the frame (cross-shard ownership-move + delivery), and a third client in another room receives nothing. +- **Given** 1k connected clients with one hot sender, **when** the + reduction budget preempts, **then** every room keeps making progress + (no starvation) on one OS thread per core, verified by TID. +- **Given** SIGTERM with clients connected, **when** the server drains, + **then** every connection gets a close frame, every fiber unwinds its + drop maps (ASan zero leaks), and the process exits 0. +- **Given** the web-app running beside chat features, **when** the + standing gates run, **then** nothing regresses — HTTP and WS share the + serve loop honestly. + +## Out Of Scope + +Message persistence/history (a `@table` an app adds if it wants — not the +workload's point); auth beyond the existing bearer mechanism; permessage +compression; binary frames beyond echo coverage; wss (TLS stays at the +proxy — the proxy story extends to WS pass-through, documented). + +## Info + +- Dependencies: the 8+11 arc (fibers + cross-shard send), the crypto + builtins fork (SHA-1 for the upgrade handshake — note: the ledger's + crypto slice lists SHA-256/512; the WS handshake specifically needs + SHA-1, so the builtin set must include it), and the framework's parse + seam (upgrade is an HTTP request until it isn't). +- Unparks on landing: the framework ledger's WebSocket/pub-sub rows and + the iteration-18 rejection note ("pub/sub REJECTED until 8/11"). + +## Proposed Solution + +Brainstorm → spec → plan after the arc's spec exists; the arc's plan and +this iteration's are written against each other (the arc names chat as +its acceptance, chat names the arc as its substrate). diff --git a/docs/stories/language-runtime-database/refine/09g-query-grammar-corpus.md b/docs/stories/language-runtime-database/refine/27-query-grammar-corpus.md similarity index 99% rename from docs/stories/language-runtime-database/refine/09g-query-grammar-corpus.md rename to docs/stories/language-runtime-database/refine/27-query-grammar-corpus.md index e6ca122..5573de4 100644 --- a/docs/stories/language-runtime-database/refine/09g-query-grammar-corpus.md +++ b/docs/stories/language-runtime-database/refine/27-query-grammar-corpus.md @@ -1,4 +1,4 @@ -# Iteration 9g — query grammar, driven by real embedded-DB corpora +# Iteration 27 — query grammar, driven by real embedded-DB corpora > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](../00-story.md). diff --git a/docs/stories/language-runtime-database/refine/14-skillhost-host-workload.md b/docs/stories/language-runtime-database/refine/28-skillhost-host-workload.md similarity index 97% rename from docs/stories/language-runtime-database/refine/14-skillhost-host-workload.md rename to docs/stories/language-runtime-database/refine/28-skillhost-host-workload.md index 12778e9..d1ed877 100644 --- a/docs/stories/language-runtime-database/refine/14-skillhost-host-workload.md +++ b/docs/stories/language-runtime-database/refine/28-skillhost-host-workload.md @@ -1,4 +1,4 @@ -# Iteration 14 — skillhost: a host-shaped workload, and the capability gaps it exposes +# Iteration 28 — skillhost: a host-shaped workload, and the capability gaps it exposes > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](../00-story.md). @@ -34,7 +34,7 @@ ## What is already expressible (verified 2026-08-16) - **The skill catalog** — `@table` with the query surface already exceeds - skillhost's in-memory SQLite `skills` table; iteration 9g's + skillhost's in-memory SQLite `skills` table; iteration 27's `docs/examples/skill-catalog` is literally this table, running. (Or a plain `map`/`multi` would do — the catalog is a lookup cache, not persistence.) - **Discovery** — `fs.exists`/`fs.list` (one level) + `fs.read_all` walk @@ -65,7 +65,7 @@ Two directions, and they are a real fork, not a detail: - **FFI as a language capability** — a way to declare and call C functions from writeonce. This is a large, doctrine-level addition (the runtime is libc-only by principle; the one sanctioned exception so far is the vendored - Ed25519, 9d). FFI would reopen the dependency-sprawl question the whole + Ed25519, 21). FFI would reopen the dependency-sprawl question the whole project is built to avoid. Likely its own spec, likely contested. - **Out-of-process model, no FFI** — drive a llama.cpp binary via `proc` (`llama-cli`) or `llama-server` over `net` + `json` (it accepts a GBNF @@ -194,6 +194,6 @@ then B and the partials narrow the gap to skillhost's real behavior. exactly the open gaps (A: out-of-process model, B: socket not stdio, C: bounded once its iteration lands, plus the fs partials) — the list is the iteration's own scoreboard. -- Reuse iteration 9g's `skill-catalog` as the catalog layer, log-watcher's +- Reuse iteration 27's `skill-catalog` as the catalog layer, log-watcher's MCP mode as the transport skeleton, and the systems stdlib for discovery and execution. diff --git a/docs/stories/language-runtime-database/refine/13-compile-time-metaprogramming.md b/docs/stories/language-runtime-database/refine/29-compile-time-metaprogramming.md similarity index 98% rename from docs/stories/language-runtime-database/refine/13-compile-time-metaprogramming.md rename to docs/stories/language-runtime-database/refine/29-compile-time-metaprogramming.md index 3b9fc89..63cb956 100644 --- a/docs/stories/language-runtime-database/refine/13-compile-time-metaprogramming.md +++ b/docs/stories/language-runtime-database/refine/29-compile-time-metaprogramming.md @@ -1,4 +1,4 @@ -# Iteration 13 — compile-time metaprogramming (derive from the class table) +# Iteration 29 — compile-time metaprogramming (derive from the class table) > Format: fiberloom `product/story-iteration-template`. Part of > [Story — one language, one runtime, one database, one binary](../00-story.md). @@ -99,7 +99,7 @@ rather than against it. - **Monomorphized generics as a general feature.** Per-type generation here is specific to the derive set, not a general generics engine. - **Deriving across the attach channel** — a client generating an encoder over - the owner's types (iterations 9c/9d). Composes later; the class-table + the owner's types (iterations 20/21). Composes later; the class-table metadata already crosses the channel's schema handshake, so the pieces are in place, but it is not this iteration's problem. - **Reopening principle 13 in any form.** If a derive appears to need runtime diff --git a/docs/superpowers/plans/2026-08-01-http-service-layer.md b/docs/superpowers/plans/2026-08-01-http-service-layer.md index 255fa82..859b514 100644 --- a/docs/superpowers/plans/2026-08-01-http-service-layer.md +++ b/docs/superpowers/plans/2026-08-01-http-service-layer.md @@ -1,6 +1,6 @@ # HTTP Service Layer Implementation Plan -> **Status: ⬜ pending** (story iteration 10) — `service` blocks route to VM methods; REST parity with the shipped Rust Stage 2 runtime. Board: [00-status.md](../../00-status.md) +> **Status: ⬜ pending** (story iteration 25) — `service` blocks route to VM methods; REST parity with the shipped Rust Stage 2 runtime. Board: [00-status.md](../../00-status.md) > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > diff --git a/docs/superpowers/plans/2026-08-20-framework-v2-memory-features.md b/docs/superpowers/plans/2026-08-20-framework-v2-memory-features.md new file mode 100644 index 0000000..89443bd --- /dev/null +++ b/docs/superpowers/plans/2026-08-20-framework-v2-memory-features.md @@ -0,0 +1,349 @@ +# Iteration 18 — framework v2 (transaction{} + cache/flags/jobs): implementation plan + +> **Status: ready to execute (2026-08-20).** Board: +> [docs/00-status.md](../../00-status.md). + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> superpowers:subagent-driven-development (recommended) or +> superpowers:executing-plans to implement this plan task-by-task. Steps +> use checkbox (`- [ ]`) syntax for tracking. +> +> **Style rule (user convention):** concept, reason, and required behavior +> in words plus verification commands only — no implementation or test +> code blocks; the executor writes the code. + +**Goal:** `transaction { }` makes multi-statement writes one WAL commit +(trap = abort), and three framework pieces ride the single process: +a TTL cache, `@table` feature flags with a cached read, and a durable +`@table` job queue drained in-process — the web-app proves the headline +(order + job enqueue, ONE commit, SIGKILL-survivable). + +**Architecture:** the engine already stages WAL batches +(`wal_append_*` → `wo_wal_commit`, per-statement today in +`database/src/db.c`'s three write cases); a transaction defers the commit +and keeps an undo log for RAM. The VM learns two internal builtins and a +transaction-flagged catch frame; the parser one keyword. Cache/flags/jobs +are pure `.wo` in the framework plus one serve-loop seam. No new opcodes, +no `.wob` version bump, GC untouched. + +**Tech Stack:** C11 libc-only (engine + VM), OCaml stdlib-only (`woc`), +pure `.wo` (framework), bash gates. + +**Spec:** [`../specs/2026-08-20-memory-db-features-design.md`](../specs/2026-08-20-memory-db-features-design.md) +(approved 2026-08-20, normative). Story: +[`18-memory-db-features.md`](../../stories/language-runtime-database/hold/18-memory-db-features.md). + +## Global Constraints + +- Branch `framework-v2` off master; commits local only, never push. +- Exit/trap doctrine unchanged: WO-E110 nested `transaction` (compile), + WO_T_DB "nested transaction" (dynamic, through a call); commit failure + is the existing WO_T_IO shape with RAM never ahead of disk. +- Gates that must stay green after every task: `just woc-test`, + `just oop-e2e`, `just deps-accept`, `just web-app`, `just log-watcher`, + `just employee` — plus the oop-accept ASan clause for anything the VM + touches. +- Framework tables carry the `wf_` prefix; policy stays app-side. +- `time.now` is wall-clock MILLISECONDS everywhere in this iteration. + +## Spec deviations, disclosed up front + +1. **`return`/`break`/`continue` crossing a `transaction { }` boundary is + rejected at compile time (new WO-E112).** The spec is silent on early + exit; commit-on-return vs abort-on-return is exactly the ambiguity a + v1 must not guess at. A later iteration may define it; today the block + has one entrance and one exit (a TRAP still aborts — that path is + defined). +2. **The cache exposes a time-injected seam** (`get_at`/`put_at` taking a + now-milliseconds argument, with `get`/`put` sugar reading `time.now`) + so the corpus fixture injects stamps instead of sleeping — the spec's + "stamps injected, not waited" made concrete. +3. **Flags are gate-proven, not corpus-proven**: a `@table` needs + `WO_DATA` and a persistent directory, which the corpus harness does + not provide; `just web-app` carries the flags checks (the spec's gate + section already put them there). +4. **The WAL gains an explicit staged-batch discard** (`wo_wal_abort`): + today a failed commit "stays staged" by contract; abort needs to drop + the batch deliberately. Same file, same batch machinery, new entry + point. + +--- + +## Task 1 — engine transactions: defer commit, undo log, abort + +**Files:** +- Modify: `database/src/db.h` (txn state on `wo_db`, three new entry + points), `database/src/db.c` (the three write cases ~lines 17–80), + `database/src/wal.h` + `wal.c` (`wo_wal_abort`), + `database/src/table.h` + `table.c` only if the replay-only fixed-id + create/index pair needs a non-static wrapper. +- Test: `runtime/test/test_txn.c` (new, mirroring the existing + `test_*.c` harness shape), wired into the runtime test recipe. + +**Interfaces:** +- Produces: `wo_db_txn_begin(db)` (0 ok; nonzero = already open — the + dynamic-nesting signal), `wo_db_txn_commit(db, wal)` (the ONE + `wo_wal_commit`; on failure the undo walk runs and the error returns), + `wo_db_txn_abort(db, wal)` (reverse undo walk + `wo_wal_abort`). + Task 3's builtins call exactly these three. + +- [ ] `wo_db` grows a transaction flag and an undo log (a growable array + of entries: op kind, class id, row id, and for update/delete a copy of + the row's bytes — `wo_row_ptr` + the class's `row_size` are the copy; + the log exists only while a transaction is open, zero cost otherwise). +- [ ] The three write cases in `wo_builtin_db`: when the flag is set, + record the undo entry BEFORE the RAM apply (update/delete pre-images; + insert records just the new id AFTER apply), still `wal_append_*`, and + SKIP the per-statement `wo_wal_commit`. Flag clear = byte-identical + behavior to today (every existing gate is the proof). +- [ ] Statement-level failures inside a transaction change nothing: a + unique violation traps before apply and stages nothing (already true — + `wo_row_insert` refuses first); a failed append keeps the pre-existing + error shape. +- [ ] Abort walks the undo log in REVERSE: inserted row → removed; + updated row → bytes restored and index entries fixed the replay way + (remove + re-add through the same engine-internal pair `wal.c` replay + uses); deleted row → re-created with its FIXED id and re-indexed (the + replay-only create), then `wo_wal_abort` discards the staged batch. +- [ ] Reads inside a transaction need no change: RAM stays applied, so + scans/point-reads see the block's own writes for free. +- [ ] `test_txn.c`: begin→insert+insert→commit = both rows + ONE wal + flush; begin→insert→abort = zero rows, next insert works; update and + delete pre-images restored on abort (indexed column included); + begin-while-open refused; abort with an empty log is a no-op. +- [ ] Run the runtime test suite + `just employee` (engine untouched when + no txn opens). Commit. + +## Task 2 — the language surface: keyword, block, WO-E110/E112 + +**Files:** +- Modify: `compiler/src/token.ml` + `lexer.ml` (KwTransaction), + `compiler/src/ast.ml` (a Transaction statement holding a body), + `compiler/src/parser.ml` (block statement + LEXICAL nesting = + WO-E110), `compiler/src/types.ml` (walk the body; E-code constants), + `compiler/src/owner.ml` (treat as a plain nested scope), + `compiler/src/dump.ml` (labels). + +**Interfaces:** +- Consumes: nothing new. +- Produces: the AST node Task 3 lowers; WO-E110 (parsing prefix, nested + block), WO-E112 (parsing prefix, `return`/`break`/`continue` whose + jump would cross the block boundary — a loop wholly INSIDE the block + keeps its own break/continue). + +- [ ] Keyword + statement parse; the body is an ordinary statement list. + A `transaction` token while one is already open (parser-tracked depth) + is WO-E110 at the inner keyword. +- [ ] WO-E112: while parsing the block, a `return` at any depth, or a + `break`/`continue` not enclosed by a loop that itself started inside + the block, names the rule ("a transaction has one exit; lift the + return out or end the block first"). +- [ ] Types/owner: the body checks exactly like a bare block — no new + typing rule (the ownership pass sees a scope; values born inside drop + inside, exactly as today). +- [ ] Compile-fail fixtures: `transaction-nested` (WO-E110), + `transaction-early-return` (WO-E112). Verify both + `just woc-test` + (dump labels) + `just oop-e2e`. Commit. + +## Task 3 — VM lowering: internal builtins + the abort-on-unwind frame + +**Files:** +- Modify: `runtime/src/wob.h` (two builtin ids in the internal range), + `runtime/src/builtin.c` (dispatch to Task 1's three entry points), + `runtime/src/vm.c` (transaction-flagged catch frame; the vm_trap walk; + unwind/rt-destroy cleanup), `compiler/src/emit.ml` (lower the + Transaction statement), `runtime/test/test_unwind.c` (frame cleanup on + a trap that leaves the whole method). + +**Interfaces:** +- Consumes: Task 1's `wo_db_txn_begin/commit/abort`, Task 2's AST node. +- Produces: the observable spec semantics — one fdatasync at the closing + brace; trap unwinding OUT aborts and keeps unwinding; `try` INSIDE the + block keeps it alive. + +- [ ] Two builtin ids the emitter emits directly from the Transaction + case (no name in any user-callable table — nothing to collide with): + begin pushes a TRANSACTION-FLAGGED catch frame (the existing TRY frame + machinery with one flag bit) and calls txn_begin (nonzero = the + WO_T_DB "nested transaction" trap); end calls txn_commit FIRST and + pops the frame only on success — a commit failure traps with the frame + still in place, so the abort path below runs and RAM is rolled back + (the "RAM never ahead of disk" rule, transaction-sized). +- [ ] vm_trap's handler search: a transaction-flagged frame is not a + handler — abort the transaction, pop it, CONTINUE searching. An inner + `try` frame sits ABOVE it and catches first (the spec's + inner-try-keeps-it-alive rule falls out of frame order, no special + case). +- [ ] Program exit / rt teardown with a transaction somehow open (a trap + that reaches main uncaught) must abort, not leak the undo log. +- [ ] Emitter: begin, body statements, end — plus the WO-E112 guarantee + from Task 2 meaning no jump ever leaves the region except a trap. +- [ ] Corpus: `run/transaction-commit` (two inserts, both rows readable + after — needs the trap corpus's WO_DATA-less shape? No: @table without + WO_DATA runs RAM-only with no WAL, which still exercises begin/commit + frames; the DURABILITY half lives in Task 6's gate where WO_DATA + exists), `run/transaction-abort` (second insert unique-traps, caught + OUTSIDE the block: first row gone too, inserts after the abort work, + process exits clean under ASan). +- [ ] `just oop-e2e` (ASan stage covers the new frames) + full battery. + Commit. + +## Task 4 — framework cache: TTL + capacity, pure `.wo` + +**Files:** +- Create: `docs/examples/writeonce-framework/store/cache.wo`. +- Test: `tests/corpus/run/cache-ttl/` (fixture copies the class inline — + corpus fixtures cannot `use` the framework; the framework file is the + same code verified by the framework's standalone compile + Task 7's + consumer build). + +**Interfaces:** +- Produces: `pub class Cache { ttl_ms: Int, cap: Int, keys: multi Text, + vals: map, stamps: map }` with `get_at(now, + key) -> ?Text`, `put_at(now, key, value)`, and `get`/`put` sugar over + `time.now` — the shape apps hold as a field on any long-lived + instance. + +- [ ] `get_at`: absent → nil; older than `ttl_ms` → remove the entry + (lazy expiry — there are no timers by design) and answer nil; live → + the value (caller-owned copy). +- [ ] `put_at`: store + stamp; when the key list exceeds `cap`, evict + OLDEST-INSERTED until within capacity (FIFO — the file states the + LRU tradeoff the spec settled). Re-putting an existing key refreshes + value + stamp without duplicating the key entry. +- [ ] Values are Text; the file says "json.encode structure into it" — + no generics exist, stated, not apologized for. +- [ ] Fixture drives injected stamps: fresh hit, expiry at exactly + ttl+1, eviction order under cap pressure, re-put refresh; ASan run. +- [ ] Framework standalone compile stays clean. Commit. + +## Task 5 — framework flags: wf_flags + cached read-through + +**Files:** +- Create: `docs/examples/writeonce-framework/store/flags.wo`. + +**Interfaces:** +- Produces: `@table(name: "wf_flags")` class `Flag { name: Text @unique, + on: Int }` (Int 0/1 — Bool columns are unproven storage, spec's call) + and `pub class Flags { loaded: Int, cache: map }` with + `read(name, default: Bool) -> Bool` and `set(name, on: Bool)`. + +- [ ] `read`: first call fills the map from the table (query by name — + the employee-proven point-read), later calls answer from the map; + absent flag → the default, uncached (so a later `set` is seen). +- [ ] `set`: update-or-insert the row, then update the map in the same + call — single process, invalidation is an assignment. Durability is + the table's (WAL), restart rebuilds via `read`. +- [ ] Framework standalone compile; behavior proven in Task 7's gate + (deviation 3). Commit. + +## Task 6 — framework jobs: wf_jobs, enqueue, JobRunner, the idle seam + +**Files:** +- Create: `docs/examples/writeonce-framework/store/jobs.wo`. +- Modify: `docs/examples/writeonce-framework/http/serve.wo` (Dispatcher + gains `fn idle()`; the serve loop calls it after `net.accept`, BEFORE + parsing the connection's first request), `app.wo` (`App` satisfies + `idle`; `jobs(take r: Jr, budget: Int)` registration; `Jr { r: + JobRunner }` wrapper, the Mw/Route pattern), `README.md` (the drain + contract + the idle-server-drains-nothing disclosure). + +**Interfaces:** +- Consumes: `transaction { }` (Tasks 1–3) only in the DEMO — enqueue + itself is an ordinary insert, composition happens in app code. +- Produces: `@table(name: "wf_jobs")` class `Job { kind: Text, payload: + Text, attempts: Int, not_before: Int }`; `pub fn enqueue(kind, + payload)`; `pub interface JobRunner { fn run(kind: Text, payload: + Text) -> Bool }`; `App.jobs(take r, budget)`; `Dispatcher.idle()`. + +- [ ] Drain (in `App.idle`): no runner registered → return immediately. + Else query up to `budget` due jobs (`not_before <= time.now`, + registration order via `take`), each inside `try`: true → `delete` + the row; false or trap → `attempts + 1` (update), row stays — + retry/backoff policy is the app's (it can rewrite `not_before` from + its own runner). +- [ ] The post-accept/pre-parse placement is the DETERMINISM the gate + needs: a job enqueued by connection A never runs before A closes, and + a kill after A's response provably leaves the row. Latency cost + (≤ budget jobs ahead of the next request) stated in the README. +- [ ] Framework standalone compile; the serve loop's existing gates + (`just web-app` current count) stay green with NO runner registered — + the seam must cost nothing. Commit. + +## Task 7 — the web-app demo + the gate + +**Files:** +- Modify: `docs/examples/web-app/main.wo` (transactional CreateOrder + + confirm runner + `GET /jobs` count + `POST /flags/:name` + the + flag-gated header on the product list), `types.wo` (nothing — wf_ + tables come from the framework), `README.md`, + `scripts/web-app-accept.sh`. + +**Interfaces:** +- Consumes: everything above, through `[deps]` exactly like every other + framework feature. + +- [ ] `CreateOrder.handle`: `transaction { insert Order {...}; + enqueue("confirm", ); }` — the headline composition, one + commit. A `Confirm` runner class answers `confirm` by printing the + order-confirmation line to stderr and returning true; registered via + `app.jobs(Jr { r: Confirm {...} }, budget)`. +- [ ] `GET /jobs` (behind the existing bearer auth): pending count as + JSON. `POST /flags/:name`: flips through `Flags.set`; the product + list answers an extra header (e.g. `x-store-banner`) while the flag + is on. +- [ ] Gate additions, in order: (a) `POST /orders` 201, then `kill -9` + the server IMMEDIATELY (no further requests), restart on the same + `WO_DATA`, then `GET /jobs` — the confirmation line appears in the + restarted server's log (the drain ran post-accept on this very + request) and the count answers 0: the job survived the kill because + it committed WITH the order; (b) `POST /flags/banner` then + `GET /products` carries the header, restart, still carries it; + (c) the standing matrix unchanged. Counts stay dynamic in the script. +- [ ] Full battery: `just web-app`, `woc-test`, `oop-e2e`, + `deps-accept`, `log-watcher`, `employee`. Commit. + +## Task 8 — docs closeout + +**Files:** +- Modify: `docs/00-status.md` (row 18 ✅ with measured results; NEXT + PLAN advances to 20/21 per the order), story `18-memory-db-features.md` + (landing banner) then `git mv` into `stories/.../done/` with links + re-pathed and VERIFIED, `docs/00-dependency-graph.md` (node classes: + 18 done; TPRMW unblocks), framework `README.md` (ledger rows: + storage-integration txn-per-request now buildable; the v2 pieces ✅), + `compiler/src/CODE-LOGIC.md` + `runtime/src/CODE-LOGIC.md` + + `database/src/CODE-LOGIC.md` (the txn seams, one paragraph each). +- [ ] Apply; run `just web-app` once more after doc edits; commit. + +## Success criteria (spec, restated as the gate reads them) + +1. Two inserts in one block: SIGKILL before the next request → both rows + after restart (gate a); a trap unwinding out → neither row and the + process keeps serving (`run/transaction-abort` + ASan). +2. The order's job runs after the NEXT accepted connection within + budget, never before the posting connection closes, and survives a + kill in between (gate a). +3. An expired or evicted cache entry answers nil with no timer having + existed (`run/cache-ttl`, stamps injected). +4. A flipped flag holds across restart (gate b). Every standing gate + green; opcode set and `.wob` format byte-identical. + +## Self-review notes + +- Spec coverage: Part A semantics → Tasks 1–3 (observable rules mapped + one-to-one; the early-exit hole closed by deviation 1); cache → T4; + flags → T5; jobs + seam → T6; demo + gate → T7; out-of-scope list + untouched. Corpus/gate split follows deviations 2–3. +- Type consistency: the three engine entry points, the two E-codes + (E110/E112), `wf_flags`/`wf_jobs`, `get_at`/`put_at`, + `JobRunner.run(kind, payload) -> Bool`, `App.jobs(take r, budget)`, + `Dispatcher.idle()` — spelled identically in every task that names + them. +- Risk, disclosed: the abort walk's index restoration is the one place + correctness is subtle (indexed column updated then aborted); Task 1's + unit test pins exactly that case before any VM work stacks on it. +- Ordering: engine (T1) before VM (T3) with the language (T2) between so + T3 has both; cache/flags (T4/T5) are independent and could land any + time, kept after the critical path so the risky work gets the freshest + attention. diff --git a/docs/superpowers/plans/2026-08-20-library-kind-internal.md b/docs/superpowers/plans/2026-08-20-library-kind-internal.md new file mode 100644 index 0000000..a67ed1b --- /dev/null +++ b/docs/superpowers/plans/2026-08-20-library-kind-internal.md @@ -0,0 +1,267 @@ +# Iteration 17 — library kind + `internal/`: implementation plan + +> **Status: ⏸ PARKED 2026-08-20** (developer directive: framework v1 work +> proceeds instead; this plan stays ready on branch `library-internal`, +> execution not started). Board: [docs/00-status.md](../../00-status.md). + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> superpowers:subagent-driven-development (recommended) or +> superpowers:executing-plans to implement this plan task-by-task. Steps use +> checkbox (`- [ ]`) syntax for tracking. +> +> **Style rule (user convention):** this plan carries concept, reason, and +> required behavior in words plus verification commands only — no +> implementation or test code blocks; the executor writes the code. + +**Goal:** `kind = "library"` in `wo.toml` makes a project checkable without +an entry, the `internal/` rule keeps a dependency's plumbing private +(WO-E108), and the framework adopts both — `just web-app` proves the public +surface unmoved. + +**Architecture:** every change is compile-time and lives in the driver +(`compiler/bin/main.ml`): the manifest reader learns one key, the manifest +build path grows a check branch, and the dep-use resolution walk in +`compile_image` grows the boundary rule. No lexer, parser, typechecker, VM, +`.wob`, or GC change — the spec's impact analysis is normative. + +**Tech Stack:** OCaml stdlib only (`woc`), bash acceptance scripts, `just`. + +**Spec:** [`../specs/2026-08-20-library-kind-internal-design.md`](../specs/2026-08-20-library-kind-internal-design.md) +(normative). Story: +[`17-library-projects-internal.md`](../../stories/language-runtime-database/17-library-projects-internal.md). + +## Global Constraints + +- Branch `library-internal`; commits local only, never push. +- OCaml stdlib only; no new executables, no new build steps. +- Absent `kind` means program — every existing project's behavior is + byte-identical; the standing gates prove it each task. +- Exit-code doctrine stays: 0 clean, 1 diagnostics, 2 usage/manifest/IO. + WO-E108 is a diagnostic (exit 1, printed by the normal collector path); + WO-E109 is a manifest error (exit 2, `woc: : error WO-E109: ...`, + the WO-E106 shape). +- `internal` matches a whole path SEGMENT of a dep-relative module path — + never a substring. +- Gates that must stay green after every task: `just woc-test`, + `just oop-e2e`, `just deps-accept`, `just web-app`, `just log-watcher`, + `just employee`. + +## Spec deviations, disclosed up front + +1. The spec put WO-E108/E109 fixtures in the compile-fail corpus. The + corpus harness (`scripts/oop-e2e.sh`) compiles a fixture DIRECTORY with + no manifest and no network — a `wo.toml` in a fixture dir would flip woc + into manifest-build mode, and WO-E108 additionally needs a fetched dep. + Both diagnostics are therefore gated in `scripts/web-app-accept.sh` + (which already builds a file:// dep chain), not the corpus. Task 5. +2. The spec says check mode "stops before image emission". The cheapest + correct implementation reuses `compile_image` whole — emission happens + in memory and the image is discarded; no artifact is written, no entry + is required (an entry-less image is already legal there, the `--emit` + precedent). Behavior matches the spec; the pipeline boundary is one + step later than the spec's wording. +3. `woc build -o ` never reads the manifest, so it resolves no + `[deps]` (pre-existing, iteration 15). The dual lib+bin case therefore + holds for libraries without `[deps]` — the framework qualifies. Recorded + here, not fixed; a dep-aware explicit build is future work. + +--- + +## Task 1 — the manifest `kind` key + WO-E109 + +**Files:** +- Modify: `compiler/bin/main.ml` (`manifest_parse` known-key table + ~line 808; `manifest_build` ~line 1023). + +**Interfaces:** +- Produces: `manifest_parse` accepts top-level `kind`; `manifest_build` + exposes the validated kind ("program" when absent) to Task 2's branch. + +- [ ] `manifest_parse`: add `kind` to the known TOP-LEVEL keys (the match + arm that today allows `name`, `version`, `description`). Nothing else in + the parser changes — the value is a quoted string like every other key. +- [ ] `manifest_build`: after the `name` check, read `kind`. Absent or + `"program"` continues to build. `"library"` is Task 2's branch (for this + task, temporarily fall through to build — the framework does not carry + the key until Task 4, so nothing observable changes). Any OTHER value + fails as a manifest error in the WO-E106 print shape with code WO-E109, + naming the given value and the two legal ones, exit 2. +- [ ] Verify by hand: a scratch project under the scratchpad with + `kind = "junk"` refuses with WO-E109 and exit 2; the same project with + `kind = "program"`, and with no `kind` line, builds as before. +- [ ] Verify nothing moved: `just woc-test && just deps-accept && just employee`. +- [ ] Commit. + +## Task 2 — library check mode + the build-error hint + +**Files:** +- Modify: `compiler/bin/main.ml` (`manifest_build`; `build_mode` no-entry + error ~line 639; `usage_msg` ~lines 42–52 and the long help's build + paragraph ~line 96). + +**Interfaces:** +- Consumes: Task 1's validated kind. +- Produces: `woc ` on a `kind = "library"` manifest runs the full + check; the behavior Tasks 4–5 verify against. + +- [ ] `manifest_build`, kind `"library"`: resolve `[deps]` and enforce the + `[runtime]` constraint exactly as build does (a library must be + checkable offline once locked), then run `compile_image ~deps dir`, + print diagnostics through the normal `finish` path on error (exit 1), + and exit 0 silently on success. No `target/` directory is created, no + file is written, no entry is required. The full pipeline runs — parse, + typecheck, interface satisfaction, ownership, GC inference — because + `compile_image` already runs it; the in-memory image is discarded + (deviation 2). +- [ ] `build_mode`, the no-entry error: when `/wo.toml` exists and + names `kind = "library"`, append the library hint to the existing + message — the project is a library; add a `main` for a demo binary or + check it with `woc `. Read the manifest only if the file exists; + malformed manifests keep failing as they do today. +- [ ] `usage_msg`: the `woc ` line notes "builds a program / checks a + library, per the manifest's `kind`"; the long help gains two sentences + on `kind` and check mode. No new flags. +- [ ] Verify by hand: scratch library project (`kind = "library"`, one + class, no `main`) — `woc ` exits 0 with no output; plant a type + error, `woc ` exits 1 with the normal diagnostic; `woc build + -o x` fails with the no-entry message plus the library hint; add a + `main`, `woc build` produces a runnable binary (dual case) while + `woc ` still only checks. +- [ ] Verify nothing moved: `just woc-test && just oop-e2e && just deps-accept`. +- [ ] Commit. + +## Task 3 — the `internal/` boundary rule (WO-E108) + +**Files:** +- Modify: `compiler/bin/main.ml` (`compile_image`'s dep-use walk, + ~lines 487–518). + +**Interfaces:** +- Consumes: the existing walk that already knows, per file, whether a dep + owns it (`owner`) and prefixes dep-internal use paths. +- Produces: WO-E108 diagnostics in the collector; Task 5 gates on the code + string. + +- [ ] In the same `List.map` over parsed files: for a file NOT owned by any + dep (the `owner = None` branch, today untouched), inspect each `Use` + whose FIRST segment names a dep in `deps` — when any LATER segment is + exactly `internal`, add a diagnostic to the collector: code WO-E108, the + use's file and `pos`, message naming the internal module path and the + dependency it belongs to (the `Diag.error` + `Collector.add` pattern at + ~line 277). Do not drop the use — the collector's has-error path already + prevents emission, and later resolution errors on the same use are + harmless duplicates suppressed by exit-on-first-report ordering as + today. +- [ ] Dep-owned files stay untouched on this path — the boundary is + consumer-only (spec §3): the dep's own `use internal` (prefixed to + `/internal` by the existing arm) must keep compiling. +- [ ] The root project's own `internal/` directories are NOT matched: the + rule keys on the first segment being a DEP name, so a root-project + `use internal` never fires it. No code needed — assert it in the Task 5 + gate instead. +- [ ] Verify by hand: temp dir pair — a dep with an `internal/` module and + a consumer importing `/internal` — compile fails exit 1 printing + WO-E108 at the `use`'s line; the dep's own file importing `internal` + compiles. +- [ ] Verify nothing moved: `just woc-test && just deps-accept && just web-app`. +- [ ] Commit. + +## Task 4 — framework reorg: adopt `kind` + `internal/` + +**Files:** +- Modify: `docs/examples/writeonce-framework/wo.toml` (add + `kind = "library"`), `app.wo` (imports), `README.md` (verification note). +- Move: `http/parse.wo` → `internal/parse.wo`, + `http/serve.wo` → `internal/serve.wo` (git mv; module becomes + `framework/internal` under a consumer, `internal` standalone). +- Not touched: `http/types.wo`, `router/router.wo`, the web-app. + +**Interfaces:** +- Consumes: Task 2's check mode (standalone verification), Task 3's rule + (what the move protects). +- Produces: the reorganized layout Task 5's gate checks against. + +- [ ] Move the two plumbing files. Same-module access dies with the move: + `internal/serve.wo` and `internal/parse.wo` now need `use http` for + `Req`/`Resp` (they shared the `http` module with `types.wo` before); + `app.wo` adds `use internal` for the serve loop and `Parsed` seam. No + declaration changes — imports only. +- [ ] `wo.toml` gains `kind = "library"` (top-level, beside `name`). +- [ ] README: replace the `--emit` verification workaround sentence with + the check-mode invocation (`woc ` — full pipeline, no entry), and + one sentence on `internal/` being unimportable by consumers. +- [ ] Verify: `compiler/_build/default/bin/woc docs/examples/writeonce-framework` + exits 0 silently — the workaround is dead. +- [ ] Verify the consumer: `just web-app` — all 14 standing checks pass + unchanged (the app imports only `framework`, `framework/http`, + `framework/router`, so NOTHING in it changes). +- [ ] Commit. + +## Task 5 — gate: three new checks in web-app-accept + +**Files:** +- Modify: `scripts/web-app-accept.sh` (after the build check, before the + serve matrix). + +**Interfaces:** +- Consumes: the `$W/fw` framework copy and `$W/app` consumer the script + already builds; Tasks 1–4's behavior. +- Produces: `just web-app` at 17 checks, the iteration's single gate. + +- [ ] Check 15 — library check mode: `woc "$W/fw"` (the framework copy, + which now carries `kind = "library"` and no `main`) exits 0 with empty + output. +- [ ] Check 16 — the boundary: copy the app to a second temp dir, append a + `use framework/internal` line to its `main.wo`, compile; assert exit 1 + and `WO-E108` in stderr. +- [ ] Check 17 — kind validation: copy the app again, set + `kind = "junk"` in its manifest, compile; assert exit 2 and `WO-E109` + in stderr. +- [ ] Renumber nothing — the script counts dynamically (`pass`/`fail`); + update only the header comment's check inventory. +- [ ] Run the full battery: `just web-app` (17/0) and the standing gates — + `just woc-test`, `just oop-e2e`, `just deps-accept`, `just log-watcher`, + `just employee`. +- [ ] Commit. + +## Task 6 — docs closeout + +**Files:** +- Modify: `docs/00-status.md` (NEXT PLAN advances; board row 17 → done + with what landed; in-progress row moves to the next order item), + story `17-library-projects-internal.md` (landing banner), + `compiler/src/CODE-LOGIC.md` (driver section: `kind`, check mode, the + boundary rule), `docs/08-project-structure.md` (framework layout gains + `internal/`), framework story `16-web-framework.md` only if it names the + `--emit` workaround. +- [ ] Apply; every claim carries its measured gate result (17/0 etc.); no + forward-looking "will". +- [ ] `just web-app` once more after the doc edits (nothing should move — + honesty check). +- [ ] Commit. + +## Success criteria (spec, restated as the gate reads them) + +1. Framework checks entry-less (`woc ` exit 0) and `woc build` on it + fails naming the library kind — gate check 15 + Task 2's hand check. +2. A consumer import of `framework/internal` is WO-E108 at the `use`; + the framework's own import stays legal — gate check 16 + Task 3. +3. `just web-app`'s original 14 checks pass unchanged after the reorg — + Task 4/5. +4. A manifest without `kind` behaves byte-identically — every standing + gate, every task. + +## Self-review notes + +- Spec coverage: §1 kind key → Task 1; §2 driver modes → Task 2; §3 rule → + Task 3; §4 diagnostics → Tasks 1–3; §5 reorg → Task 4; §6 gate → Task 5; + out-of-scope list untouched by any task. Corpus-fixture clause replaced + by deviation 1; emission-boundary wording by deviation 2; dual-with-deps + limit by deviation 3. +- Type consistency: the only cross-task names are the two code strings + (WO-E108, WO-E109), the manifest key `kind`, and the module path + `framework/internal` — spelled identically in every task. +- Risk, disclosed: moving serve/parse breaks same-module visibility they + silently enjoyed beside `types.wo`; Task 4 names the exact import each + file must gain, and the standalone check catches any miss before the + gate runs. diff --git a/docs/superpowers/plans/2026-08-20-shard-fiber-arc.md b/docs/superpowers/plans/2026-08-20-shard-fiber-arc.md new file mode 100644 index 0000000..78738af --- /dev/null +++ b/docs/superpowers/plans/2026-08-20-shard-fiber-arc.md @@ -0,0 +1,227 @@ +# The 8+11 concurrency arc — implementation plan (staged) + +> **Status: STAGES 1 AND 2 COMPLETE 2026-08-20** (branch `concurrency-arc`, +> T1–T4 landed + the `docs/examples/fibers` demo and its `just fibers` +> gate, 8/0). Stages 2–3 pending. Execution deviations, disclosed: +> (1) the reduction budget decrements at loop BACK-EDGES ONLY, after the +> jump lands — the spec's "same three sites as the GC" wording had a +> livelock at budget 1 (pre-instruction save re-executes the jump into +> the same decrement; pinned by test_fiber's exact round-robin); +> (2) T4 landed io_uring-FIRST per the amended spec (park.c: raw ring, +> POLL_ADD/TIMEOUT, WO_IO=uring|epoll, epoll fallback) and the battery +> ran green on BOTH backends plus LW_SOAK over parked I/O; +> (3) a partial net.write's progress crosses the park via park_wr_at — +> the retry protocol needed write-offset state the spec never named. +> Stage-2 deviations: (4) the inbox is a MUTEX-guarded list + eventfd, +> not the spec's lock-free MPSC ring — rings arrive when 22 measures the +> mutex; (5) WO-E222 applies to EVERY spawn/send (round-robin placement +> makes any actor potentially remote — compile time cannot see the +> shard); (6) the deterministic corpus pins WO_SHARDS=1, and the +> multi-shard truth (output SETS, TSan, both I/O backends) lives in the +> fibers gate — the spec's narrowed-determinism rule made operational; +> (7) two TSan-caught races fixed (late-init memset vs concurrent push; +> wake-efd read outside the lock) and one teardown SEGV (routed frees +> during teardown are now no-ops: arenas die wholesale). +> Board: [docs/00-status.md](../../00-status.md). + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> superpowers:subagent-driven-development (recommended) or +> superpowers:executing-plans task-by-task. Steps use checkbox syntax. +> +> **Style rule (user convention):** concept, reason, required behavior in +> words plus verification commands only — the executor writes the code. + +**Goal:** the spec's three stages — fibers on one shard, shards with +ownership-moving sends, the transparent DB actor — each landing with +every standing gate green before the next begins. + +**Architecture:** see the spec (normative): +[`../specs/2026-08-20-shard-fiber-arc-design.md`](../specs/2026-08-20-shard-fiber-arc-design.md). +Stories: [8](../../stories/language-runtime-database/refine/08-shard-actor-runtime.md) · +[11](../../stories/language-runtime-database/refine/11-fibers.md). + +**Tech Stack:** C11 libc-only (`wovm`), OCaml stdlib-only (`woc`), bash +gates; TSan added to the corpus harness at stage 2. + +## Global Constraints + +- Branch `concurrency-arc` off master; commits local only, never push. +- A stage does not start until the previous stage's FULL battery is + green: `just woc-test`, `just oop-e2e` (ASan stage included), + `just deps-accept`, `just web-app`, `just log-watcher`, + `just employee`. +- Stage 1 changes NO observable behavior for single-fiber programs — + every existing fixture byte-identical; that is the refactor's proof. +- New E-codes: WO-E221 (spawn needs `receive(msg: M)`), WO-E222 + (traced-containing cross-shard send). `.wob` format: spawn/send lower + to new BUILTIN ids — no new opcodes; the version bumps ONLY if the + actor-type field encoding forces it (decide in T3, disclose). + +--- + +## Stage 1 — fibers on one shard + +### Task 1 — the fiber context extraction (pure refactor) + +**Files:** `runtime/src/vm.h`, `runtime/src/vm.c`, `runtime/src/gc.c` +(`vm_gc_roots`), `runtime/src/main.c`, `runtime/test/*` (compile fixes +only). + +**Interfaces:** produces `wo_fiber` (registers, frames, depth, catch +stack, caught error, resume pc/state) and `wo_vm` holding module + rt + +current-fiber + a fiber list; everything else consumes `vm->cur`. + +- [x] Move the five interpreter-state fields into the fiber struct; the + vm allocates fiber 0 (main) at init; `vm_run`/`wo_vm_call`/unwind/ + trap paths read through the current-fiber pointer. +- [x] `vm_gc_roots` iterates ALL fibers' registers+frames (one fiber + today — the loop is the change). +- [x] Verify: full battery byte-identical (this task has NO functional + change to hide behind). Commit. + +### Task 2 — run queue + reduction budget + +**Files:** `runtime/src/vm.c`, `runtime/src/vm.h`. + +**Interfaces:** produces fiber states (RUNNABLE/PARKED/DONE), a FIFO +queue, `WO_REDUCTIONS` (default 4000), and a scheduler loop around the +dispatch loop; consumes Task 1's context switch (save/restore = swap the +current-fiber pointer). + +- [x] Budget decrements — AMENDED to back-edges only (deviation 1); originally the GC-safepoint sites (NEW, CALL, + backward-JMP); zero → re-queue + switch. One fiber = no observable + change (it re-queues to itself). +- [x] A trap unwinding out of a non-main fiber kills that fiber alone + (drops run, uncaught-trap report to stderr, program lives); main's + return ends the program and unwinds every remaining fiber through the + stop path (drop-clean). +- [x] C unit test (`runtime/test/test_fiber.c`): two hand-built fibers + interleave deterministically under a tiny budget; kill-on-main-return + leaks nothing under ASan. Full battery. Commit. + +### Task 3 — spawn / send / `actor M` (language + same-shard delivery) + +**Files:** `compiler/src/{token,lexer,ast,parser,types,owner,emit,dump}.ml`, +`runtime/src/{wob.h,builtin.c,vm.c}`, corpus fixtures. + +**Interfaces:** produces the `actor M` field-type shape (parametric like +`multi T`), `spawn Cls { ... }` expression (M inferred from Cls's +`receive`; WO-E221 otherwise), `send(addr, msg)` (msg typed M, owner +pass TRANSFERS it — sender's later use is the existing use-after-move +diagnostic), and runtime delivery: an actor is a fiber that sleeps +between messages, mailbox FIFO, one `receive` call per message. + +- [x] Type/AST/parse: `actor M` in field positions; spawn/send keywords + (grep the corpus for identifier collisions first, the standing + keyword discipline). +- [x] Owner: send's message argument is a transfer (the take machinery); + spawn's ctor literal moves its field values (existing ctor rules). +- [x] Emit + runtime: two builtin ids; spawn creates the actor fiber + + returns the address as a scalar word; send enqueues + wakes. + Addresses are copyable scalars; send-to-dead = silent drop (v1, + spec'd). +- [x] Corpus: `run/actor-echo` (spawn, send, receive mutates actor + state, deterministic output), `run/actor-move` (sender's reuse after + send = compile-fail fixture, actually its own + `compile-fail/send-after-move`), `compile-fail/spawn-no-receive` + (WO-E221). ASan. Full battery. Commit. + +### Task 4 — parking builtins on the shard's io_uring loop (AMENDED) + +**Files:** `runtime/src/builtin.c` (net/time cases), `runtime/src/vm.c` +(the per-shard ring the scheduler idles in), `runtime/src/sysio.c`. + +**Interfaces:** produces PARKED fibers keyed by ring `user_data`; +consumes Task 2's states. AMENDED 2026-08-20 (developer directive + +the linux reference project): the loop is **io_uring in readiness mode** +(POLL_ADD for fds, TIMEOUT for sleeps; resume re-executes the now-ready +builtin), with the startup probe + `WO_IO=uring|epoll` override and the +epoll path kept as the PORTABILITY fallback (seccomp'd containers deny +io_uring) — CI runs the fiber tests on both. Raw syscalls, 5.1-safe ops +only; the ring layout goes in the binding doc. 23 later rides the SAME +per-shard ring for WAL WRITE+FSYNC chains, and the ops-mode ladder +(reads/writes as ring ops with completion results) is its own follow-up +task after T4 proves the loop. + +- [x] Stop composes: the stop flag wakes every parked fiber into the + unwind path (the existing stop story, per fiber). +- [x] Proven live instead of a bespoke C test: the fibers demo (part 2) + the full battery over parked I/O (log-watcher's mcp loop, web-app's matrix) on BOTH backends + LW_SOAK; originally: fiber A parked in sleep, fiber B progresses, one TID; + A resumes on time. Corpus: `run/fiber-park-net` if expressible + without a peer (loopback pair via the existing net builtins — the + log-watcher mcp fixtures prove the shape); otherwise the unit test + carries it, disclosed. +- [x] Full battery + framework/web-app gates (serve loop untouched — + single fiber per process today, degenerate path). Commit. **Stage 1 + complete: board + stories note it.** + +## Stage 2 — shards + +### Task 5 — shard threads + per-shard runtimes + +**Files:** `runtime/src/{vm.h,vm.c,main.c,obj.c,gc.c}`, +`compiler/bin/main.ml` (manifest `[runtime] shards`). + +- [x] One pinned pthread per shard, each with its own full `wo_rt`; + count = cores default, `WO_SHARDS`/manifest override; N=1 must be + byte-identical to stage 1 (the escape hatch and the proof). +- [x] Program mode: main runs on shard 0; with no spawns the other + shards idle at zero cost (parked on their eventfds). +- [x] Full battery at N=1 AND at default cores (no cross-shard sends + exist yet — the threads just have to not break anything). Commit. + +### Task 6 — cross-shard send + traced rejection + home-routed frees + +**Files:** `runtime/src/{vm.c,obj.c,gc.c}` (MPSC rings, eventfd wake, +home-free ring), `compiler/src/types.ml` (WO-E222 via the +transitively-traced check), corpus + TSan. + +- [x] Round-robin spawn placement; pointer handoff; header shard id + routes the eventual free home (drained per tick, debug-asserted). +- [x] WO-E222 at compile time for traced-containing sends (reuse the + gc_may-shaped fixpoint the inference already computes). +- [x] The actor corpus under ASan AND TSan: zero races/leaks; output-SET + assertions (cross-shard nondeterminism is honest, spec'd). +- [x] Full battery at default cores. Commit. **Stage 2 complete.** + +## Stage 3 — the DB actor + closing the arc + +### Task 7 — transparent DB RPC + +**Files:** `runtime/src/builtin.c` (db cases marshal when not on shard +0), `database/src/` untouched (the engine never learns), `runtime/src/vm.c` +(request/reply parking). + +- [ ] Non-owner DB builtins marshal statement + args to shard 0, park, + resume with materialized reply; `transaction { }` travels as one unit + (18's staged batch stays owner-side). +- [ ] `just employee` + `just web-app` at default cores, answers + byte-identical to N=1 (iteration 8's criterion 4, the arc's headline + proof). Commit. + +### Task 8 — the arc's closeout + +- [ ] 22's benchmark re-run (or its minimal precursor if 22 has not + landed: the employee read/write loop timed) single- vs multi-shard; + numbers recorded in the stories. +- [ ] Stories 8 + 11 landing banners; board rows; graph node classes; + framework README ledger rows that the arc unblocks (streaming etc. + stay ⏸ until their own slices — the arc UNBLOCKS, it does not build + them); CODE-LOGIC files (vm fiber model, shard/mailbox model, DB RPC). +- [ ] Full battery once more after doc edits. Commit. + +## Success criteria + +The spec's four, verbatim — stage 1's determinism + ASan-clean unwind, +stage 2's TSan-clean moves + WO-E222, stage 3's byte-identical +multi-shard gates + recorded 22 delta, and the language having grown +exactly `spawn`/`send`/`actor M`. + +## Self-review notes + +- Spec coverage: Part A → T1–T4; Part B → T5–T6; Part C → T7–T8; + diagnostics T3 (E221) / T6 (E222); out-of-scope untouched. +- The riskiest surgery (T1) is deliberately a PURE refactor with a + byte-identical battery as its only claim — functional change never + hides inside it. +- Names used consistently: `wo_fiber`, `vm->cur`, WO-E221/E222, + `WO_REDUCTIONS`, `WO_SHARDS`, `actor M`, `spawn`/`send`. diff --git a/docs/superpowers/specs/2026-08-18-web-framework-design.md b/docs/superpowers/specs/2026-08-18-web-framework-design.md index 5c7226d..c5aa9bf 100644 --- a/docs/superpowers/specs/2026-08-18-web-framework-design.md +++ b/docs/superpowers/specs/2026-08-18-web-framework-design.md @@ -7,8 +7,8 @@ library, HTTP/1.1 behind a TLS-terminating reverse proxy, and (C) the parked HTTP/2 path. Three sub-projects; A and B are the fundable ones, C is a recorded successor. -**Relates to:** iteration 10 (`service` blocks — this framework becomes their -lowering target, not a rival), iterations 8/9f/11 (the concurrency work h2c +**Relates to:** iteration 25 (`service` blocks — this framework becomes their +lowering target, not a rival), iterations 8/23/11 (the concurrency work h2c waits for), `docs/plan/discarded.md` (FFI reject row — load-bearing here). ## Decisions locked during brainstorming @@ -17,7 +17,7 @@ waits for), `docs/plan/discarded.md` (FFI reject row — load-bearing here). | --- | --- | | TLS | **Proxy-terminated** (nginx/caddy). Browsers get TLS+ALPN+h2 from the proxy; the framework speaks HTTP/1.1 (later h2c) behind it. Zero TLS in the language or runtime. Direct-serving TLS is a separate future iteration and would be judged against the FFI/libc-only doctrine then, not now. Homegrown TLS is refused outright: a decade of side-channel and certificate-validation subtleties makes it a security liability, not a milestone. | | Dependencies | **`wo.toml [deps]` + git fetch.** A real (mini) package manager: exact-rev git dependencies, a lockfile, a per-project cache. No registry, no semver solving. | -| HTTP/2 | **v1 is HTTP/1.1 keep-alive.** h2's payoff is multiplexing, which a single blocking thread cannot exploit; h2c lands as its own iteration after shards (8) / io_uring (9f) / fibers (11). Behind the proxy, browsers see h2 from day one regardless. | +| HTTP/2 | **v1 is HTTP/1.1 keep-alive.** h2's payoff is multiplexing, which a single blocking thread cannot exploit; h2c lands as its own iteration after shards (8) / io_uring (23) / fibers (11). Behind the proxy, browsers see h2 from day one regardless. | | Handler model | **Structural interfaces, not closures.** The language has no function values by doctrine; a route handler is a class satisfying a `Handler` interface, dispatched by ICALL — which works on today's runtime and is checked by WO-E205. | | Incubation | Framework is born at `docs/examples/writeonce-framework/`; the consuming app at `docs/examples/web-app/` imports it **through the `[deps]` mechanism** (a local git URL), so the whole import chain is exercised by the sample. Extraction to `github.com/shoneyj/` later is a `git subtree split`, not a redesign. | @@ -136,7 +136,7 @@ routes, restart-persistence check, SIGTERM. ## C. HTTP/2 (h2c) — parked successor -After iterations 8 (shards) / 9f (io_uring) / 11 (fibers): h2c framing + +After iterations 8 (shards) / 23 (io_uring) / 11 (fibers): h2c framing + HPACK, either natively (needs a bytes/buffer type with cheap slicing — that type rides with this iteration, not v1) or via nghttp2-in-runtime (a doctrine decision to re-argue then, with the TweetNaCl precedent and the libc-only diff --git a/docs/superpowers/specs/2026-08-20-library-kind-internal-design.md b/docs/superpowers/specs/2026-08-20-library-kind-internal-design.md new file mode 100644 index 0000000..c27de6d --- /dev/null +++ b/docs/superpowers/specs/2026-08-20-library-kind-internal-design.md @@ -0,0 +1,159 @@ +# Iteration 17 — library projects and dependency privacy: design + +> **Status: ⏸ PARKED 2026-08-20** (developer directive: framework v1 work +> proceeds instead; spec + plan stay ready on branch `library-internal`). +> Approved before parking. Decisions were settled in +> [the iteration](../../stories/language-runtime-database/17-library-projects-internal.md) +> (four forks + impact analysis); this spec makes them buildable. The plan +> follows after review. Board: [docs/00-status.md](../../00-status.md). +> +> Per repo convention this spec carries concept, reason, and required +> behavior in words only — no implementation code. + +## Goal + +A project can say it is a library, and a dependency can keep modules to +itself. Concretely: `docs/examples/writeonce-framework` declares +`kind = "library"` in its `wo.toml`, `woc ` on it typechecks the whole +project with no entry required (the iteration-16 `--emit` workaround is +deleted), its parser and serve-loop plumbing move under `internal/` where +the web-app cannot import them, and `just web-app` proves nothing public +broke. + +## Background (the two gaps, from iterations 15/16) + +1. A project without `fn main` cannot be checked: manifest presence forces + build mode, which errors "no `main` entry point found". The framework is + verified today through `woc --emit` — a wart. +2. `pub` is module-public with no dep-private tier: `parse_request` is + exactly as importable by the consuming app as `Handler`. Nothing marks + "this module is the library's own business". + +## Settled decisions (normative, from the iteration) + +- Library-ness is manifest-declared: top-level `kind` key, values + `"program"` (the default when absent) and `"library"`; any other value is + a manifest error. +- Privacy is Go's `internal/` directory rule, applied at the `[deps]` + boundary only: a consumer cannot `use` a dependency module whose path + contains an `internal` segment; inside the dependency the same import + stays legal. +- Lib+bin duality is allowed: a library may carry an entry-shaped `main`; + `kind = "library"` changes only the DEFAULT action of `woc `. +- Inherited from Go, explicitly not checked: an internal type may appear in + a public signature. The consumer can hold and pass such a value but + cannot import the module to name its type. Library author's smell to + avoid, not a diagnostic. + +## Design + +### 1. The manifest `kind` key + +One top-level key in `wo.toml`, read where the manifest is already parsed +(the driver's manifest reader in `compiler/bin/main.ml`). Absent means +program — every existing project keeps its behavior. A value other than +`"program"` or `"library"` is rejected with the new WO-E109, naming the +value and the two legal ones. The key is meaningful only in the project +being invoked; a DEPENDENCY's `kind` is read but not enforced in v1 +(iteration 15 already never uses a dep's `main`, so a program consumed as a +dep already behaves as a library — recorded, not policed). + +### 2. Driver modes + +- `woc ` on a program: unchanged — build, requiring `main`. +- `woc ` on a library: CHECK mode — the full pipeline runs (parse, + typecheck, interface satisfaction, borrow/ownership pass, GC inference) + over the whole project including its `[deps]`, and stops before image + emission. No entry is required. Exit 0 with no output on success, 1 with + diagnostics otherwise — a green check must mean exactly what a green + build means, minus the artifact. Deps are fetched/locked the same as + build mode (a library must be checkable offline once locked). +- `woc build -o ` on a library WITH a `main`: builds the binary + — the dual case (demo/self-test). Without a `main`: the existing + no-entry error, extended to name the kind so the message explains itself + ("this project is a library; add a main or check it with woc "). +- `--emit`, dump flags, `--update-deps`: unchanged; they never required an + entry or already carry their own rules. + +### 3. The `internal/` rule + +Where: the dep-use resolution step in `compile_image` — the same place +iteration 15 prefixes dep module paths — because that is the only spot that +knows which file belongs to which project. The rule: a `use` in a file that +does NOT belong to dependency X, naming a module of dependency X whose +dep-relative path contains a segment exactly equal to `internal`, is +rejected with WO-E108 at that `use`, naming the dependency and the module. +Files INSIDE dependency X importing the same module are untouched. Modules +are directory-shaped (`module_of_multi`: dep name + relative directory), so +"segment" means a path component — `framework/internal` and anything under +it, never a substring match (a module named `internals_x` is not caught). + +Scope notes, normative: the rule fires only across the `[deps]` boundary +(decision 3 — dep-boundary-only; Go's subtree rule is a recorded possible +tightening). The root project's own `internal/` directories are legal to +import from anywhere inside the root project. Transitive deps stay +rejected by iteration 15's WO-E106, so dep-to-dep imports cannot occur. + +### 4. Diagnostics (error catalog additions) + +- **WO-E108** — dep-internal module imported across the `[deps]` boundary. + Points at the offending `use`, names the dependency and the internal + module, and says the module is internal to that dependency. +- **WO-E109** — invalid manifest `kind` value. Names the given value and + the two legal ones. +- The no-entry build error gains the library wording described in §2; no + new code, better message. + +### 5. Framework reorg (the proof by use) + +- `wo.toml` gains `kind = "library"`. +- `http/parse.wo` and `http/serve.wo` move to `internal/` (module + `framework/internal` when consumed). They are plumbing the web-app never + imports: the request parser, the carry-state record, the serve loop, the + `Dispatcher` seam. +- `http/types.wo` (Req/Resp + builders), `router/router.wo` + (Handler/Middleware/Route/Mw), and `app.wo` (App) stay where they are — + the public surface does not move. +- Intra-framework imports update to the new module path; the web-app + changes NOTHING — it already imports only `framework`, `framework/http`, + `framework/router`. +- The framework README's verification note replaces the `--emit` + workaround with the check-mode invocation. + +### 6. Acceptance gate + +Extend `scripts/web-app-accept.sh` (it already builds the framework remote +and the consuming app): the standing 14 checks stay, plus (a) check mode — +`woc` on the framework copy exits 0 with no entry present; (b) privacy — a +temp copy of the app with one added `use framework/internal` fails +compile and the output names WO-E108; (c) kind validation — a temp +manifest with a junk `kind` fails naming WO-E109. Compiler-side, the +corpus grows compile-fail fixtures for WO-E108/E109 and a run fixture is +not needed (no runtime behavior exists to pin — the VM is untouched by +design). Standing gates (`just woc-test`, `just deps-accept`, +`just oop-e2e`, samples) stay green. + +## Out of scope (recorded, deliberate) + +Go's full subtree rule; a `[lib]` manifest section; `pub(lib)`-style +keyword visibility; manifest export allowlists; multiple named binaries +per project (`cmd/` convention); enforcing a dep's own `kind`; internal +types in public signatures as a diagnostic; any VM, `.wob`, or GC change +(impact analysis in the iteration: visibility is compile-time name +resolution; libraries compile whole-program into the consumer's image; +GC inference stays whole-program and app usage may promote dep classes — +intended). + +## Success criteria + +1. **Given** the framework with `kind = "library"` and no `main`, **when** + `woc ` runs, **then** it exits 0 having run the full pipeline, and + `woc build` on it fails with the message naming the kind. +2. **Given** the web-app importing `framework/internal`, **when** it + compiles, **then** WO-E108 points at the `use` and names the framework; + the framework's own files importing it stay legal. +3. **Given** the reorganized framework, **when** `just web-app` runs, + **then** all standing checks pass unchanged — the public surface did + not move. +4. **Given** any existing project with no `kind` key, **when** it builds, + **then** nothing changed — absent means program. diff --git a/docs/superpowers/specs/2026-08-20-shard-fiber-arc-design.md b/docs/superpowers/specs/2026-08-20-shard-fiber-arc-design.md new file mode 100644 index 0000000..f26319d --- /dev/null +++ b/docs/superpowers/specs/2026-08-20-shard-fiber-arc-design.md @@ -0,0 +1,243 @@ +# The 8+11 concurrency arc — shards, fibers, actors: design + +> **Status: spec, awaiting review (2026-08-20).** The arc's decisions were +> settled in [iteration 8](../../stories/language-runtime-database/refine/08-shard-actor-runtime.md) +> (refined) and the brainstorm of 2026-08-20 (this document's Decisions). +> Covers iterations 8 AND 11 as one arc; iteration 24 (chat) is its +> acceptance workload and gets its own spec after this one. The plan +> follows after review. Board: [docs/00-status.md](../../00-status.md). +> +> Per repo convention: concept, reason, and required behavior in words +> only — no implementation code. + +## Goal + +The runtime scales past one core without ever showing a thread, a lock, +or a colored function: pinned thread-per-core shards whose only +communication is ownership-moving messages, cooperative fibers above them +preempted by reduction budget, blocking builtins that park instead of +block, and a database that stays single-writer by being runtime plumbing +on its owner shard. One `spawn`/`send` surface; app code byte-identical +at one shard or sixteen. + +## Decisions (settled; the spec builds on them) + +From the 2026-08-20 refinement: the DB is an actor on an owner shard +(never user-visible); 8+11 are one arc; one unified address surface; +chat (19) is the driving workload; order 22 → arc → 19 → 23. + +From the arc brainstorm (2026-08-20): + +1. **All cores by default.** `WO_SHARDS` / `[runtime] shards = N` + override (N=1 is the debug/serial escape hatch); the arc's landing + criterion is every standing gate green AT the default. Brave by + choice: the gates become the stress test. +2. **Typed receive.** An actor is any class with `fn receive(msg: M)` — + structural, the Handler doctrine, no blocking-receive keyword, no new + interface to implement. `spawn Cls { ... }` infers M from the class's + `receive` signature and returns an **`actor M`** address (a + parametric builtin type exactly like `multi T`/`map`). + `send(addr, msg)` compile-checks that msg is an M and MOVES it. +3. **Transparent DB RPC.** insert/update/delete/query builtins executed + on a non-owner shard marshal the statement to the owner shard and + PARK the calling fiber until the reply; results come back + materialized (they already do). App code never changes; the fourth + iteration-8 criterion (byte-identical answers single- or multi-shard) + is the proof. +4. **Fibers first, then shards.** Stage 1 ships fibers on one shard + (framework's fiber-per-connection lands early, no threads); stage 2 + ships pinned threads + cross-shard send; stage 3 ships the DB RPC and + re-runs 22. Each stage gates green independently. + +## Part A — the fiber runtime (stage 1, one shard) + +### The fiber context + +A fiber IS the interpreter state `wo_vm` already isolates: the register +window (`WO_STACK_SLOTS`), the frame stack (`WO_MAX_FRAMES`), the catch +stack, the caught-error slot, and a resume point. Stage 1 extracts those +fields into a fiber context; `wo_vm` keeps the module, the runtime +(arena/GC/db — per SHARD, not per fiber), the current-fiber pointer, and +a FIFO run queue. Cost, stated: a context is ~42 KiB at today's +constants (32 KiB registers + frames + catches), so 1k fibers ≈ 42 MiB — +acceptable v1, arena-allocated; segmented/growable windows are recorded +future work, not v1. + +### Scheduling + +- FIFO run queue, no priorities (iteration 11's out-of-scope holds). +- **Preemption by reduction budget**: the budget decrements at the + EXISTING safepoint sites (NEW, CALL, backward-JMP — the same places + 7b's GC polls, one counter check added, no new instrumentation); at + zero the fiber re-queues and the next runs. Default budget 4000 + reductions, `WO_REDUCTIONS` overrides. Deterministic: same program, + same schedule. +- `main` is fiber 0. When it returns, the program's exit value is + main's; every other fiber — runnable or parked — is UNWOUND through + the drop machinery (the SIGTERM/stop story, reused verbatim): ASan + zero leaks is the criterion. A trap that unwinds out of a non-main + fiber kills that fiber alone (its drops run, its error goes to stderr + the uncaught-trap way); the program does not die. +- GC: every fiber's registers and frames are roots — `vm_gc_roots` + iterates all contexts, not just the live one. Parked and queued + fibers pin their values exactly like the running one. + +### spawn / send (same-shard in stage 1) + +- `spawn Cls { fields }` — an expression; Cls must declare + `fn receive(msg: M)` where M is a class, record, or union type + (**WO-E221** otherwise, naming what's missing); the ctor literal's + fields are the actor's state, moved in (the take-push shape). Result + type `actor M`. +- `send(addr, msg)`: msg must type as M (existing assign/param + machinery) and be OWNED (the owner pass's transfer rules; a borrowed + or copy-in-hand value is the existing WO-E3xx shape). After the send + the sender's binding is dead — compile-time move, the iteration-8 + criterion. +- Delivery: the runtime calls the actee's `receive` with the message, + one message at a time per actor, on the actor's home shard, as its + fiber's next work item. An actor is a fiber that sleeps between + messages; its mailbox is the FIFO the runtime owns. +- Same-heap sends hand the pointer over — no copy, no serialization; + the move made aliasing impossible at compile time. +- `actor M` values are plain copyable words (an address), storable in + fields/containers like Int — sending TO a dead actor is a silent drop + of the message v1 (Erlang's shape; a delivery-receipt story is future + work, recorded). + +### The I/O plane: io_uring-first (AMENDED 2026-08-20, developer directive) + +The original wording made epoll the v1 loop with io_uring as a +benchmark-gated swap. That framing is REVERSED by directive, and the +[linux reference project](../../plan/exploration/linux/07-io_uring.md) +already pointed here: io_uring is the successor to epoll + libaio for +BOTH the storage engine's WAL path and the socket path — one event loop. + +- **One io_uring per shard thread** is THE event loop, serving three op + families as the arc progresses: (1) fiber parking — `IORING_OP_POLL_ADD` + for parked net fds, `IORING_OP_TIMEOUT` for sleeps, the mailbox eventfd + registered the same way (stage 2's wake); (2) WAL durability — 23's + WRITE + FSYNC(DATASYNC) chains ride the SAME ring, `user_data` the + commit correlation, the shard executing while the ring drains; (3) the + data path — multishot accept/recv/send for iteration 24's connection + fan-in, LAST, because it changes resume semantics. +- **The readiness→ops ladder, staged honestly:** T4 uses the ring in + READINESS mode (POLL_ADD/TIMEOUT — resume re-executes the now-ready + builtin, the same retry contract epoll would have had, so the builtin + layer barely changes). Submitting the reads/writes THEMSELVES as ring + ops (completion carries the result, no retry) is the follow-up step — + bigger builtin surgery, staged after T4 proves the ring loop. +- **Kernel floor and ops discipline:** io_uring is Linux 5.1+, mature + 5.11+; v1 uses only 5.1-safe ops (POLL_ADD, TIMEOUT, WRITE, FSYNC). + Multishot accept (5.19) / multishot recv (6.0) are recorded + optimizations for 19's spec, gated on the probe, never assumed. +- **The fallback is PORTABILITY, not preference:** a startup probe + (`io_uring_setup`, fall back on ENOSYS/EPERM — seccomp'd containers + routinely deny io_uring) selects the ring or the epoll+blocking-fsync + path; `WO_IO=uring|epoll` overrides so CI proves BOTH paths on one + kernel (this subsumes 23's `WO_WAL_MODE`). io_uring is the design; + epoll exists so the binary runs everywhere — the project's premise. +- Raw `io_uring_setup`/`io_uring_enter` syscalls, libc-only (23's settled + fork, now arc-wide); the mmap'd ring layout goes in the binding doc, + normative, exactly like the WAL format. +- `net.accept`/`net.read`/`net.write` and `time.sleep` PARK the calling + fiber against the ring; the shard runs other fibers; completion + re-queues the parked one. With exactly one fiber the ring wait IS the + blocking call — same code path, program mode is the one-fiber case. +- `fs.*` stays genuinely blocking in v1 (local disk, bounded); its reads + become ring ops in the ops stage if 22's numbers ask; recorded. +- The stop story composes: the stop signal wakes the ring (the existing + self-pipe/eventfd trick), every parked fiber unwinds — the SIGTERM + drain criterion extends to fibers for free. +- The framework consequence (its own slice, after stage 1): the serve + loop spawns a fiber per connection and the close-when-idle policy + dies; that lands with iteration 24's spec, not this one. + +### Fiber context growth (recorded improvement, post-stage-1) + +A fiber context is ~42 KiB today (fixed register window + frame array). +Unlike native-stack green threads (Go's grow-by-copy must rewrite every +pointer into the old stack), this VM's "stack" is register-INDEXED, not +address-based — a context is relocatable by construction, so growth is +an allocate-larger + memcpy with zero pointer fixups. Start-small +(~4 KiB) growable contexts are therefore cheap to add and take fiber +counts from thousands toward millions; scheduled after stage 1, before +iteration 24's 1k-connection target if measurement asks. + +## Part B — shards (stage 2) + +- One pinned pthread per shard; shard count = cores by default + (`WO_SHARDS`/manifest override). Each shard owns a full `wo_rt`: + arena, traced list, GC — 7b's collector is per-shard by construction, + so no global pause exists to remove. +- Cross-shard `send`: an MPSC mailbox ring per shard plus an eventfd to + wake an idle shard's epoll loop (the c-runtime exploration's proven + pair). The message POINTER crosses; the object's home shard is stamped + in its header (there since iteration 2); its eventual free routes back + to the allocation-home arena (a small home-free ring per shard, + drained at the tick). +- What may cross: OWNED values whose class transitively contains no + TRACED field — the `gc_may`-shaped fixpoint the runtime already + computes, surfaced at compile time: sending a value whose type is or + contains an inferred-traced class is **WO-E222**, naming the traced + class and why it is traced. Aliased graphs never cross heaps. +- `spawn` placement: round-robin across shards by default; no placement + argument in v1 (recorded future work if 22 shows a need). The + language never names a shard. +- Determinism criterion narrows honestly: single-shard runs stay + deterministic (stage 1's property); cross-shard interleaving is + nondeterministic by nature — the corpus asserts OUTPUT SETS and + ownership invariants under TSan, not byte-identical transcripts. + +## Part C — the DB actor + serving (stage 3) + +- The engine and WAL live on shard 0 (the owner). DB builtins executed + on the owner run exactly today's code. On any other shard they + marshal the statement into a runtime message, park the fiber, and + resume with the materialized reply. `transaction { }` (iteration 18) + marshals as one unit — the staged batch stays owner-side, semantics + unchanged. +- The listener: `net.listen` + `net.accept` stay on the shard that + calls them v1 (the framework serves from one accept loop; accepted + connections' fibers stay on that shard). Distributing accept + (SO_REUSEPORT per shard) is 19/23-adjacent future work the benchmark + must justify — recorded, not built. +- Stage 3 closes the arc: `just employee`/`just web-app` run + multi-shard byte-identical (criterion 4), the actor corpus runs under + ASan+TSan, and 22 re-runs with the before/after recorded. + +## Diagnostics (new) + +- **WO-E221** — `spawn` on a class with no `receive(msg: M)` (or an M + that is not a class/record/union). +- **WO-E222** — cross-shard send of a traced (or traced-containing) + type; names the class and the inference reason. Compile-time; stage 1 + same-shard sends are exempt (same heap, aliasing is GC's problem and + GC handles it). +- Existing machinery covers the rest: send-of-borrowed is the owner + pass's transfer rules; send-type mismatch is the parameter boundary. + +## Out of scope (arc-wide, restated) + +Fiber migration across shards; priorities/timers/structured concurrency +(recipe-box, later `.wo` libraries); async/await (permanently rejected); +cross-shard transactions/2PC; SO_REUSEPORT accept distribution; +segmented fiber stacks; delivery receipts / actor supervision trees +(future story); program-mode API changes (none — one fiber is the +degenerate case). + +## Success criteria + +1. **Stage 1:** thousands of fibers on one shard, one hot loop among + them — everything progresses (starvation-free fixture), output + deterministic; a parked-fiber shutdown unwinds ASan-clean; a fiber + blocked in `net.read` resumes while the shard served others, one TID. +2. **Stage 2:** the actor corpus under ASan+TSan — zero races, zero + leaks; an owned send compiles as a move (sender's later use is a + compile error); a traced-containing send is WO-E222; frees route + home (asserted in debug builds). +3. **Stage 3:** `just employee` and `just web-app` green at the + all-cores DEFAULT with byte-identical answers; 22's benchmark re-run + with the delta recorded; every standing gate green. +4. The language grew exactly `spawn`, `send`, and the `actor M` type — + no async/await, no locks, no thread ever visible.