diff --git a/docs/00-status.md b/docs/00-status.md index 36d79ad..5e58fbd 100644 --- a/docs/00-status.md +++ b/docs/00-status.md @@ -65,10 +65,11 @@ routing/security gaps is **framework v1**, tracked item-by-item in the (βœ…/πŸ”Ά/⬜/⏸/πŸ”§ per feature β€” timeouts and Unix sockets need `net` runtime seams, crypto hashes need C builtins since the language has no bitwise operators, streaming/cancellation park behind 8/11). The memory-rich -features are **framework v2** = iteration 18 (spec APPROVED 2026-08-20, -plan next): TTL cache, @table flags, durable job queue with -drain-on-request, `transaction { }` over the WAL's staged batch. After 18, -the order resumes at 20/21. Edges: [00-dependency-graph.md](00-dependency-graph.md). +features are **framework v2** = iteration 18 (⏸ HELD 2026-08-21 with spec +approved + plan authored intact): TTL cache, @table flags, durable job +queue with drain-on-request, `transaction { }` over the WAL's staged +batch. The pending order is now the concurrency chain (see *Pending* +below). Edges: [00-dependency-graph.md](00-dependency-graph.md). --- @@ -168,24 +169,26 @@ that sequences its tasks. Read one, approve, then the next starts. | 6 | [Program mode + stdlib](stories/language-runtime-database/done/06-program-mode-stdlib.md) | βœ… (the surface log-watcher uses) | | 7 | [log-watcher proof](stories/language-runtime-database/done/07-logwatcher-proof.md) | βœ… **landed 2026-08-15** β€” executable, not merely compilable: zero ASan leaks in all three modes, SIGTERM ends parked syscalls, fds flat, `LW_SOAK` gate; `just log-watcher` 7/0 | | 7b | [Inferred GC + mark-sweep](stories/language-runtime-database/done/07b-inferred-gc-mark-sweep.md) | βœ… **landed 2026-08-18** β€” `@gc` gone (WO-E104), GC-ness inferred, RC replaced by incremental mark-sweep, `.wob` v4; supersedes iteration 2's RC memory model | -| 8 | [Shard-actor runtime](stories/language-runtime-database/refine/08-shard-actor-runtime.md) | ⬜ | +| 8 | [Shard-actor runtime](stories/language-runtime-database/refine/08-shard-actor-runtime.md) | πŸ”„ arc stages 1+2 landed 2026-08-20 (branch concurrency-arc); stage 3 (transparent DB actor) = **first in the concurrency chain** | | 9 | [Database engine](stories/language-runtime-database/done/09-database-engine.md) | πŸ”„ engine complete (storage/WAL/indexes/insert-update-delete); reads land with 9b | | 9b | [`@table`, relations, query](stories/language-runtime-database/done/09b-table-relations-query.md) | πŸ”„ query surface + relations + FK done (branch query-surface); group-by parked | | 19 | [Float + Bytes](stories/language-runtime-database/done/19-missing-scalar-types.md) | βœ… **landed 2026-08-20** β€” `.wob` v5: Float constant tag, field kinds 6/7, opcodes 34-41 (IEEE-quiet f64), builtins 70-83. Full stack: literals, arithmetic, `@table` column, WAL bit-exact replay, json fractions in / shortest-round-trip out, `?Float` reserved-NaN nil, total-order index (NaN last, `-0.0` == `+0.0`), Bytes + base64. No implicit Int/Float mixing (WO-E201); `float`/`trunc` are the only bridges. Proof: web-app price is a real Float (`{"price":9.99}`), `just web-app` 23/0; corpus 103/0 | -| 20 | [Cross-program tables](stories/language-runtime-database/refine/20-cross-program-tables.md) | πŸ”„ channel done (branch ipc-attach); manifest+binding pending | -| 21 | [Keypair attach auth](stories/language-runtime-database/refine/21-keypair-attach-auth.md) | πŸ”„ crypto+handshake done (branch keypair-auth); manifest pending | -| 22 | [Durability, throughput, scale](stories/language-runtime-database/refine/22-durability-throughput-scale.md) | ⬜ needs a spec first | -| 23 | [io_uring group-commit](stories/language-runtime-database/refine/23-io-uring-commit.md) | ⬜ after 8 + 22 | -| 27 | [Query grammar corpus](stories/language-runtime-database/refine/27-query-grammar-corpus.md) | ⬜ needs a spec first | -| 10 | [HTTP service layer](stories/language-runtime-database/25-http-service.md) | ⬜ | Hold | -| 11 | [Fibers](stories/language-runtime-database/refine/11-fibers.md) | ⬜ | Hold | -| 12 | [Blue-green deploy](stories/language-runtime-database/26-blue-green-deploy.md) | ⬜ | Hold | -| 13 | [Compile-time metaprogramming](stories/language-runtime-database/refine/29-compile-time-metaprogramming.md) | ⬜ needs a spec first | -| 14 | [skillhost host workload](stories/language-runtime-database/refine/28-skillhost-host-workload.md) | ⬜ gaps recorded (branch query-grammar found skillhost needs no new query grammar); each gap a candidate iteration | +| 11 | [Fibers](stories/language-runtime-database/refine/11-fibers.md) | πŸ”„ stages 1+2 landed 2026-08-20 with 8 (`just fibers` 8/0); closes with the arc's stage 3 | +| 22 | [Durability, throughput, scale](stories/language-runtime-database/refine/22-durability-throughput-scale.md) | ⬜ needs a spec first β€” second in chain, after arc stage 3 | +| 31 | [Actor lifecycle](stories/language-runtime-database/refine/31-actor-lifecycle.md) | ⬜ needs a spec first β€” third in chain (story written 2026-08-21) | +| 24 | [chat: WebSocket workload](stories/language-runtime-database/refine/24-chat-websocket-workload.md) | ⬜ fourth in chain β€” the arc's acceptance; after 31 | +| 23 | [io_uring group-commit](stories/language-runtime-database/refine/23-io-uring-commit.md) | ⬜ last in chain, after stage 3 + 22 | +| 20 | [Cross-program tables](stories/language-runtime-database/hold/20-cross-program-tables.md) | ⏸ hold (2026-08-21); channel done (branch ipc-attach keeps its manifest) | +| 21 | [Keypair attach auth](stories/language-runtime-database/hold/21-keypair-attach-auth.md) | ⏸ hold (2026-08-21); crypto+handshake done (branch keypair-auth keeps its manifest) | +| 25 | [HTTP service layer](superpowers/plans/2026-08-01-http-service-layer.md) | ⏸ hold (2026-08-21) β€” story file removed; the plan doc remains | +| 26 | [Blue-green deploy](stories/language-runtime-database/hold/26-blue-green-deploy.md) | ⏸ hold (2026-08-21) | +| 27 | [Query grammar corpus](stories/language-runtime-database/hold/27-query-grammar-corpus.md) | ⏸ hold (2026-08-21) | +| 28 | [skillhost host workload](stories/language-runtime-database/hold/28-skillhost-host-workload.md) | ⏸ hold (2026-08-21); gaps recorded (branch query-grammar found skillhost needs no new query grammar) | +| 29 | [Compile-time metaprogramming](stories/language-runtime-database/hold/29-compile-time-metaprogramming.md) | ⏸ hold (2026-08-21) | | 15 | [deps: `wo.toml [deps]`](stories/language-runtime-database/done/15-deps-package-manager.md) | βœ… **landed 2026-08-18** (branch web-framework): [deps] inline tables, git-binary fetch, wo.lock pinning, offline-when-locked, --update-deps, WO-E106/E107; `just deps-accept` 8/0 | | 16 | [web framework](stories/language-runtime-database/done/16-web-framework.md) | βœ… **landed 2026-08-19** β€” writeonce-framework (HTTP/1.1 + router + Handler/Middleware) consumed by web-app through [deps]; h2c parked (Β§C) behind 8/23/11. **v1 polish landed 2026-08-20** (branch framework-v1): get/post/put/delete_ helpers, 405+Allow, HEAD, Logging middleware, set_header; `just web-app` 16/0; fixed the interp-borrowed-field emitter crash en route. **Auth-in-core landed 2026-08-20**: http/auth.wo (Bearer/Basic, ct_eq, req.principal), web-app dogfoods BearerAuth, gate 17/0 | | 17 | [library projects + `internal/`](stories/language-runtime-database/done/17-library-projects-internal.md) | βœ… **landed 2026-08-20** β€” `kind = "library"` in `wo.toml` (default `program`, so every existing manifest is byte-identical; unknown value = WO-E109 exit 2); `woc ` on a library runs the FULL pipeline entry-less and writes nothing, retiring iteration 16's `--emit` workaround; the no-entry build error names the kind; lib+bin dual works. Go's `internal/` rule as **WO-E108** at the consumer's own `use`, dep-boundary-only β€” the library imports its own interior freely. Framework reorganized: `internal/{parse,serve}.wo` behind the line, `http/form.wo` split out to keep `media_type`/`form_values` public. Driver-only change; VM/`.wob`/GC untouched. `just web-app` **26/0** (3 new checks), every standing gate unchanged | -| 18 | [framework v2: memory-rich features](stories/language-runtime-database/hold/18-memory-db-features.md) | πŸ”„ **spec APPROVED 2026-08-20, plan next** ([spec](superpowers/specs/2026-08-20-memory-db-features-design.md)): TTL cache + @table flags + durable job queue (drain-on-request) + `transaction { }` over the WAL's staged batch; pub/sub REJECTED until 8/11 | +| 18 | [framework v2: memory-rich features](stories/language-runtime-database/hold/18-memory-db-features.md) | ⏸ hold (2026-08-21); spec approved + plan authored, both held intact ([spec](superpowers/specs/2026-08-20-memory-db-features-design.md), [plan](superpowers/plans/2026-08-20-framework-v2-memory-features.md)): TTL cache + @table flags + durable job queue (drain-on-request) + `transaction { }` over the WAL's staged batch; pub/sub REJECTED until 8/11 | --- @@ -360,53 +363,54 @@ The C proving-ground work (`exploration/c-runtime/`, phases A–F: 859k reads/s, ## Pending -### Implementation order (re-sequenced 2026-08-20 β€” code-review pass) +### Implementation order (re-sequenced 2026-08-21 β€” concurrency chain) -Replaces the framework-goal ordering. Basis: the verified findings in -[`00-code-review.md`](00-code-review.md) β€” measure before optimizing, close -correctness holes before adding surface, stop stacking features on -unmeasured ground. IDs below are post-renumber; the authoritative table with -per-row reasoning is -[`00-story.md`](stories/language-runtime-database/00-story.md). +Everything still pending IS the runtime-concurrency chain. Basis: the +2026-08-20 code-review pass (measure before optimizing, close correctness +holes before adding surface), amended 2026-08-21 by developer decision: +**stage 3 before 22** β€” correctness first, then one benchmark campaign +covers single- and multi-shard. The authoritative table with per-row +reasoning is [`00-story.md`](stories/language-runtime-database/00-story.md). -Dependency rules that still force the shape: 23 explicitly after 8 + 22; -21's plan folds into 20's; 26 only after 9 + 25; 11 rides 8's shard -scheduler; h2c parked behind 8/23/11. +Dependency rules that force the shape: 23 after stage 3 + 22 (the ring is +the arc's, the baseline is 22's); 24 after 31 (chat is dishonest without +lifecycle); h2c stays parked behind the chain; the held tail keeps its own +precedence notes for resumption. -1. **22** β€” the measurement backbone, and now first: it has never run, so - every performance claim on this project is unsourced. No - `bench/baseline.json`, no `just db-bench`; `runtime/bench/` is the - retired C prototype's harness. -2. **8+11 stage 3** β€” transparent DB RPC, then 22 re-run for the - concurrency delta. Reframed as a correctness fix: worker VMs are - zero-initialized, so a DB statement off the primary traps `WO_T_DB`. - A multi-shard program that touches the database is broken today. -3. **30** (new) β€” observability, CI, fuzz: runtime counters + a profiler - hook, 22's harness run per change instead of by hand, a fuzz target on - the parser and `.wob` loader. No iteration covered any of this. -4. **19** β€” Float + Bytes; small, and it gates 24 (WS frames) and the - crypto fork (digests). -5. **31** (new) β€” actor lifecycle: request/response (`send` is one-way and - callers `sleep` to await), bounded mailboxes (the FIFO only grows), - actor death/supervision, timers beyond `time.sleep`. -6. **24** β€” chat, the arc's acceptance; honest only after 19 + 31. -7. **23** β€” io_uring group-commit; explicitly after 8 + 22. -8. **25** β€” HTTP service layer; `service` blocks lower onto the framework - instead of a parallel stack. -9. **18** β€” framework v2 (transaction{} + cache/flags/jobs); spec APPROVED - 2026-08-20 but **demoted from first**: more surface on a framework with - one consumer, and its cache stores `Text` because there are no generics. -10. **27, then 26** β€” query grammar from corpora (likely collapses to - "confirm `len(query)` + add `exists`"), then blue-green. -11. **20 then 21** β€” demoted hard: new distribution surface while there is - no TLS, no crypto primitives, and the multi-shard DB still traps. The - half-done branches (ipc-attach, keypair-auth) keep their manifests. -12. **28, then 29 + parked drain** β€” skillhost is no longer the driving - workload; then metaprogramming (spec first), group-by, ADT roster, - WO-E225, held by the 2026-08-08 scope directive. +1. **8+11 stage 3** β€” transparent DB actor. A correctness fix, not an + optimization: worker VMs are zero-initialized, so a DB statement off + the primary traps `WO_T_DB` β€” a multi-shard program touching the + database is broken today. Plan of record: + [`2026-08-20-shard-fiber-arc.md`](superpowers/plans/2026-08-20-shard-fiber-arc.md) + (stages 1+2 landed 2026-08-20, branch `concurrency-arc`). +2. **22** β€” the measurement backbone: restart-persistence proof + baseline + benchmark (durable + RAM-only), single- AND multi-shard in one + campaign, plus the stage-2 mutex-inbox number (rings only if the mutex + costs). It has never run β€” no `bench/baseline.json`, no `just db-bench`; + the arc's stages 1+2 delta is recorded retroactively. +3. **31** β€” actor lifecycle + ([story](stories/language-runtime-database/refine/31-actor-lifecycle.md), + written 2026-08-21): request/response (`send` is one-way and callers + `sleep` to await), bounded mailboxes (the FIFO only grows), actor + death/supervision, timers beyond `time.sleep`. +4. **24** β€” chat, the arc's acceptance; honest only after 31 (19 landed + 2026-08-20 β€” Bytes carries the frames). +5. **23** β€” io_uring group-commit; the WAL's WRITE+FSYNC chains ride the + arc's per-shard ring (T4); after 22's baseline β€” the payoff, measured. + +**30** β€” observability, CI, fuzz: named 2026-08-20, still row-only (no +story file); slots in when scheduled β€” nothing in the chain depends on it. + +⏸ **Held** (2026-08-21, developer decision): 18, 20, 21, 25, 26, 27, 28, +29 β€” stories in +[`hold/`](stories/language-runtime-database/hold/) (25's story file +removed; its [plan doc](superpowers/plans/2026-08-01-http-service-layer.md) +remains). Half-done branches (ipc-attach, keypair-auth) keep their +manifests. βœ… **17** β€” landed 2026-08-20 (unparked and executed): `kind = "library"`, check mode, and the `internal/` dep boundary (WO-E108). Driver-only. +βœ… **19** β€” landed 2026-08-20: Float + Bytes, `.wob` v5. ### Language track β€” sequenced, on the critical path diff --git a/docs/stories/language-runtime-database/00-story.md b/docs/stories/language-runtime-database/00-story.md index 2d27a31..d897cb7 100644 --- a/docs/stories/language-runtime-database/00-story.md +++ b/docs/stories/language-runtime-database/00-story.md @@ -71,6 +71,20 @@ adding surface, and stop stacking features on unmeasured ground. or timers behind `spawn`/`send`. - **18, 20, 21 demoted.** All three add surface; none answer a named gap. +RE-SEQUENCED 2026-08-21 (third pass β€” the concurrency chain). Everything +still pending IS the runtime-concurrency chain; order: +**stage 3 β†’ 22 β†’ 31 β†’ 24 β†’ 23**. Changes from the second pass: + +- **The arc's stage 3 moves ahead of 22** β€” correctness before + measurement: a multi-shard program touching the database traps + `WO_T_DB` today, and fixing that first lets one benchmark campaign + cover single- and multi-shard honestly. +- **31 has its story file** ([refine/31-actor-lifecycle.md](refine/31-actor-lifecycle.md)); + 30 stays a row until it is scheduled. +- **Holds landed** (developer decision, 2026-08-21): 18, 20, 21, 26, 27, + 28, 29 moved to `hold/`; 25's story file removed (its plan doc remains + in superpowers). 19 landed 2026-08-20. + | Seq | # | Iteration | Delivers | | --- | --- | --- | --- | | 1 | 1 | [Principles doc](done/01-principles-doc.md) | `docs/00-principles.md` β€” the doctrine page every later slice links back to | @@ -85,21 +99,21 @@ adding surface, and stop stacking features on unmeasured ground. | 10 | 9b | [`@table`, relations, query](done/09b-table-relations-query.md) | `@table` real storage; `ref`/`backlink`/`multi`; compiler-checked queries | | 11 | 15 | [deps: `wo.toml [deps]`](done/15-deps-package-manager.md) | exact-rev git deps + `wo.lock` + `.wo-deps`; flat-only, offline once locked | | 12 | 16 | [web framework](done/16-web-framework.md) | the `.wo` framework v1 (router, middleware, auth, all three body hooks) consumed via `[deps]` | -| 13 | 22 | [Durability, throughput, scale](refine/22-durability-throughput-scale.md) | restart-persistence proof, benchmarks, ~1M rows β€” the baseline the arc and 23 sign against. **Promoted to first pending (was seq 18)**: it has never run, so every performance claim on this project is currently unsourced. *(was 9e)* | -| 14 | 8+11 | [Shard-actor runtime](refine/08-shard-actor-runtime.md) Β· [Fibers](refine/11-fibers.md) | THE ARC (stages 1+2 landed 2026-08-20: fibers/budget/actors/io_uring plane; pinned shards, envelope sends, home-routed frees, WO-E222); stage 3 = transparent DB RPC + 22 re-run. **Stage 3 is a correctness hole, not an optimization**: worker shards are zero-initialized, so a DB statement off the primary traps `WO_T_DB`. | +| 13 | 8+11 | [Shard-actor runtime](refine/08-shard-actor-runtime.md) Β· [Fibers](refine/11-fibers.md) | THE ARC (stages 1+2 landed 2026-08-20: fibers/budget/actors/io_uring plane; pinned shards, envelope sends, home-routed frees, WO-E222); stage 3 = transparent DB RPC. **Promoted above 22 (2026-08-21): stage 3 is a correctness hole, not an optimization** β€” worker shards are zero-initialized, so a DB statement off the primary traps `WO_T_DB`. | +| 14 | 22 | [Durability, throughput, scale](refine/22-durability-throughput-scale.md) | restart-persistence proof, benchmarks, ~1M rows β€” the baseline the arc and 23 sign against. It has never run, so every performance claim on this project is currently unsourced; runs after stage 3 so one campaign covers single- and multi-shard, plus the stage-2 mutex-inbox number. *(was 9e)* | | 15 | 30 | Observability, CI, fuzz *(no story file yet)* | **NEW** β€” runtime counters + a profiler hook, 22's harness wired to run per change instead of by hand, and a fuzz target on the parser and `.wob` loader. The whole proof-maturity gap had no iteration to point at. | | 16 | 19 | [Float + Bytes](done/19-missing-scalar-types.md) | **LANDED 2026-08-20** β€” `.wob` v5; the full stack: IEEE-quiet f64 through literals/VM/@table/WAL/json + Bytes as the binary carrier, no implicit mixing, total-order indexes. Unblocks 24 (WS frames) and the crypto fork (digests). *(was 20)* | -| 17 | 31 | Actor lifecycle *(no story file yet)* | **NEW** β€” request/response (today `send` is one-way and callers `sleep` to await), bounded mailboxes with backpressure (today the FIFO just grows), actor death/supervision, and timers beyond `time.sleep`. 24 cannot be written honestly without these. | +| 17 | 31 | [Actor lifecycle](refine/31-actor-lifecycle.md) | request/response (today `send` is one-way and callers `sleep` to await), bounded mailboxes with backpressure (today the FIFO just grows), actor death/supervision, and timers beyond `time.sleep`. 24 cannot be written honestly without these. *(story written 2026-08-21)* | | 18 | 24 | [chat: WebSocket workload](refine/24-chat-websocket-workload.md) | the arc's acceptance: WS upgrade + frames (SHA-1 via crypto fork, Bytes via 19), rooms/broadcast, 1k clients, drain-clean. *(was 19)* | | 19 | 23 | [io_uring group-commit](refine/23-io-uring-commit.md) | WAL WRITE+FSYNC chains on the arc's per-shard rings; fsync fallback kept (after 22 + the arc). *(was 9f)* | -| 20 | 25 | [HTTP service layer](25-http-service.md) | `service` blocks lower onto the framework (after 9b + 20 by their own precedence notes). *(was 10)* | +| 20 | 25 | [HTTP service layer](../../superpowers/plans/2026-08-01-http-service-layer.md) | `service` blocks lower onto the framework (after 9b + 20 by their own precedence notes). **HELD 2026-08-21** β€” story file removed; the plan doc remains. *(was 10)* | | 21 | 18 | [framework v2: memory-rich features](hold/18-memory-db-features.md) | spec+plan approved: TTL cache, @table flags, durable job queue, `transaction { }` over the WAL's staged batch. **Demoted from seq 14**: more surface on a framework with one consumer, and the cache still stores `Text` because there are no generics | -| 22 | 27 | [Query grammar corpus](refine/27-query-grammar-corpus.md) | grow the query grammar from real corpora; likely collapses to "confirm `len(query)` + add `exists`"; precedes 28. *(was 9g)* | -| 23 | 26 | [Blue-green deploy](26-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback (plan authored after 9 + 25). *(was 12)* | -| 24 | 20 | [Cross-program tables](refine/20-cross-program-tables.md) | attach to a running program's database over local IPC; owner stays the single writer (channel half-built). **Demoted from seq 16**: new distribution surface while there is no TLS, no crypto, and the multi-shard DB still traps. *(was 9c)* | -| 25 | 21 | [Keypair attach auth](refine/21-keypair-attach-auth.md) | program identity is a keypair; mutual challenge–response at attach (crypto half-built; plan folds into 20's). **Demoted with 20** β€” and it needs crypto primitives that do not exist. *(was 9d)* | -| 26 | 28 | [skillhost host workload](refine/28-skillhost-host-workload.md) | host-shaped driving workload naming runtime gaps β€” demoted with the framework goal. *(was 14)* | -| 27 | 29 | [Compile-time metaprogramming](refine/29-compile-time-metaprogramming.md) | `@derive(...)` from class-table metadata; held with the parked drain by the 2026-08-08 scope directive. *(was 13)* | +| 22 | 27 | [Query grammar corpus](hold/27-query-grammar-corpus.md) | grow the query grammar from real corpora; likely collapses to "confirm `len(query)` + add `exists`"; precedes 28. *(was 9g)* | +| 23 | 26 | [Blue-green deploy](hold/26-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback (plan authored after 9 + 25). *(was 12)* | +| 24 | 20 | [Cross-program tables](hold/20-cross-program-tables.md) | attach to a running program's database over local IPC; owner stays the single writer (channel half-built). **Demoted from seq 16**: new distribution surface while there is no TLS, no crypto, and the multi-shard DB still traps. *(was 9c)* | +| 25 | 21 | [Keypair attach auth](hold/21-keypair-attach-auth.md) | program identity is a keypair; mutual challenge–response at attach (crypto half-built; plan folds into 20's). **Demoted with 20** β€” and it needs crypto primitives that do not exist. *(was 9d)* | +| 26 | 28 | [skillhost host workload](hold/28-skillhost-host-workload.md) | host-shaped driving workload naming runtime gaps β€” demoted with the framework goal. *(was 14)* | +| 27 | 29 | [Compile-time metaprogramming](hold/29-compile-time-metaprogramming.md) | `@derive(...)` from class-table metadata; held with the parked drain by the 2026-08-08 scope directive. *(was 13)* | | βœ… | 17 | [library projects + `internal/`](done/17-library-projects-internal.md) | **LANDED 2026-08-20** β€” `kind = "library"` + entry-less check mode (retires the `--emit` workaround) and Go's `internal/` rule as WO-E108 at the consumer's `use`; driver-only, VM/GC untouched. `just web-app` 26/0 | @@ -131,11 +145,9 @@ list, and a pointer to the plan document that already sequences its tasks. nothing MVC-scale β€” and iteration 17 (library kind + `internal/`) is **parked** with spec + plan ready on branch `library-internal`. The v1 slices landed 2026-08-20 (`just web-app` 21/0 after polish, auth, - form, multipart). Implementation order for everything still pending: - **18 (spec approved)** β†’ 20/21 β†’ 22 β†’ 8 β†’ 23 β†’ 11 (+ h2c unparks) β†’ - 10 β†’ 12 β†’ 27 β†’ 14 β†’ 13 + parked drain; 17 parked, slots anywhere after - 16 on directive. The iterations table above carries this order - row-by-row; edges live in + form, multipart). The implementation-order list this note once carried + is superseded β€” the iterations table above is the authority + (re-sequenced 2026-08-20 twice, then 2026-08-21); edges live in [`docs/00-dependency-graph.md`](../../00-dependency-graph.md). - **Iteration 7b (inserted 2026-08-11)** sits after the critical path deliberately: it delays nothing on the log-watcher line, and it must precede diff --git a/docs/stories/language-runtime-database/refine/08-shard-actor-runtime.md b/docs/stories/language-runtime-database/refine/08-shard-actor-runtime.md index 89309e0..58ff6e8 100644 --- a/docs/stories/language-runtime-database/refine/08-shard-actor-runtime.md +++ b/docs/stories/language-runtime-database/refine/08-shard-actor-runtime.md @@ -10,8 +10,18 @@ > workload is **iteration 24 (chat: WebSocket pub/sub)**. The pre-existing > plan (`docs/superpowers/plans/2026-08-01-shard-actor-vm-runtime.md`) > predates inferred GC (7b), the unified surface, and the DB decision β€” -> it is a source of ideas, NOT the plan of record; the arc starts with a -> fresh brainstorm β†’ spec β†’ plan. +> it is a source of ideas, NOT the plan of record. +> +> **RE-SEQUENCED 2026-08-21** (developer decision): the brainstorm β†’ +> spec β†’ plan happened. The arc's plan of record is +> [`2026-08-20-shard-fiber-arc.md`](../../../superpowers/plans/2026-08-20-shard-fiber-arc.md) +> (spec: [`2026-08-20-shard-fiber-arc-design.md`](../../../superpowers/specs/2026-08-20-shard-fiber-arc-design.md)), +> and **stages 1+2 LANDED 2026-08-20** on branch `concurrency-arc` +> (T1–T6, seven disclosed deviations recorded in the plan). Remaining +> scope: **stage 3, the transparent DB actor** β€” a correctness fix, not +> an optimization: worker VMs are zero-initialized, so a DB statement +> off the primary shard traps `WO_T_DB`. Concurrency-chain order: +> **stage 3 β†’ 22 β†’ 31 β†’ 24 β†’ 23**. ## Settled decisions (2026-08-20) @@ -33,8 +43,11 @@ 3. **Driving workload: chat** (iteration 24) β€” rooms, broadcast, N concurrent WebSocket clients, one binary. The arc's acceptance is the chat sample's, not only synthetic corpora. -4. **Order: 22 β†’ the 8+11 arc β†’ 23.** The io_uring write path waits for - the arc (its batch boundary is the shard tick) and for 22's baseline. +4. **Order β€” SUPERSEDED 2026-08-21.** Originally 22 β†’ the 8+11 arc β†’ 23; + in fact the arc's stages 1+2 landed before 22 ever ran (accepted + deviation β€” the before/after delta is owed and lands as 22's + multi-shard pass). Current order: **stage 3 β†’ 22 β†’ 31 β†’ 24 β†’ 23**. + 23 still waits for the arc's tick boundary and 22's baseline. ## Goals @@ -102,9 +115,12 @@ ## Proposed Solution -Fresh brainstorm β†’ spec β†’ plan for the WHOLE arc (8+11), staged: the -pinned-worker scheduler + shard-stamped heaps + MPSC mailboxes + the -unified spawn/send surface and the traced-send rejection; fibers on that -scheduler (part 2's document); the DB-actor migration; then iteration 24 -proves it. The 2026-08-01 plan is reference material for the mailbox and -heap-stamping shapes only. +Execute stage 3 of the plan of record +([`2026-08-20-shard-fiber-arc.md`](../../../superpowers/plans/2026-08-20-shard-fiber-arc.md)): +the DB-actor migration β€” engine calls off the owner shard become message +sends with parked replies, closing the `WO_T_DB` hole. Stages 1+2 +(scheduler, fibers, unified spawn/send, WO-E222 traced-send rejection, +cross-shard envelopes with home-routed frees) landed 2026-08-20. After +stage 3: 22 measures, then iteration 24 proves the arc. Actor lifecycle +(request/response, backpressure, supervision, timers) is deliberately +NOT the arc's scope β€” it is [iteration 31](31-actor-lifecycle.md). diff --git a/docs/stories/language-runtime-database/refine/11-fibers.md b/docs/stories/language-runtime-database/refine/11-fibers.md index 3984fa9..13ab2b3 100644 --- a/docs/stories/language-runtime-database/refine/11-fibers.md +++ b/docs/stories/language-runtime-database/refine/11-fibers.md @@ -20,9 +20,21 @@ > time delivery, main-return reap, fiber-trap isolation, and parked > `net`/`time` builtins on the io_uring-first per-shard I/O plane > (`WO_IO=uring|epoll`, epoll fallback proven). Demonstrated by -> `docs/examples/fibers` (`just fibers` 8/0). The shard-context criteria -> (TID assertions, cross-shard sends, blue-green drain reuse) close with -> the arc's stage 2. +> `docs/examples/fibers` (`just fibers` 8/0). +> +> **STAGE-2 SUBSTANCE LANDED 2026-08-20** (branch `concurrency-arc`, +> T5–T6): pinned thread-per-core shards, envelope sends with ownership +> move, round-robin placement, home-routed frees, WO-E222 on EVERY +> spawn/send (placement makes any actor potentially remote), TID-verified +> shard context. Deviations disclosed in the plan of record +> ([`2026-08-20-shard-fiber-arc.md`](../../../superpowers/plans/2026-08-20-shard-fiber-arc.md)): +> the inbox is a mutex-guarded list + eventfd (rings arrive only if +> iteration 22 measures the mutex as a cost), the deterministic corpus +> pins `WO_SHARDS=1`, two TSan races and one teardown SEGV fixed. +> +> **RE-SEQUENCED 2026-08-21**: what remains for the chain is the arc's +> stage 3 (transparent DB actor β€” [iteration 8](08-shard-actor-runtime.md)), +> then measurement. Order: **stage 3 β†’ 22 β†’ 31 β†’ 24 β†’ 23**. ## Goals @@ -90,15 +102,20 @@ matches the stdlib posture). - Vision origin: [blue-green vision Β§3](../../../plan/exploration/blue-green-vm/00-vision.md); iteration 8's scheduler is the substrate this extends. -- Open questions to settle in the spec β€” REDUCED 2026-08-20: the spawn - surface is settled (the unified actor address, iteration 8 decision 2). - Still open for the arc's spec: budget size and check granularity, - parked-fiber drop semantics, run-queue fairness (FIFO v1), and how a - parked fiber's borrow state interacts with the shard's GC safepoints. +- Open questions β€” REDUCED AGAIN 2026-08-21: the spawn surface, budget + size/granularity (back-edge accounting β€” see the plan's livelock + deviation), run-queue fairness (FIFO), and parked-fiber drop semantics + (fiber-trap isolation + main-return reap) are all settled by the landed + implementation. Still open: how a parked fiber's borrow state interacts + with the shard's GC safepoints β€” tracked for stage 3 / the collector's + next pass. Lifecycle surface (request/response, backpressure, + supervision, timers) is [iteration 31](31-actor-lifecycle.md)'s, not + this story's. ## Proposed Solution -- No implementation plan exists yet β€” this iteration starts with the - brainstorming β†’ spec β†’ writing-plans chain (the superpowers path every - prior iteration followed), then executes that plan. The research note - above is the brainstorm's entry material. +- The plan exists and its fiber stages are landed: + [`2026-08-20-shard-fiber-arc.md`](../../../superpowers/plans/2026-08-20-shard-fiber-arc.md) + (stages 1+2 complete 2026-08-20). This story closes when the arc's + stage 3 lands and iteration 22 records the delta; the chat workload + ([iteration 24](24-chat-websocket-workload.md)) is the proof. diff --git a/docs/stories/language-runtime-database/refine/22-durability-throughput-scale.md b/docs/stories/language-runtime-database/refine/22-durability-throughput-scale.md index 3c433e2..861486d 100644 --- a/docs/stories/language-runtime-database/refine/22-durability-throughput-scale.md +++ b/docs/stories/language-runtime-database/refine/22-durability-throughput-scale.md @@ -11,6 +11,17 @@ > iteration's benchmark and showing the number moved the right way. > > **No spec exists yet.** The forks in *Info* are genuine decisions. +> +> **RE-SEQUENCED 2026-08-21** (developer decision): runs AFTER the arc's +> stage 3 β€” the transparent DB actor is a correctness hole (a multi-shard +> program touching the database traps `WO_T_DB` today), and fixing it +> first lets ONE benchmark campaign cover single- and multi-shard +> honestly. The arc's stages 1+2 landed 2026-08-20 unmeasured; their +> delta is recorded retroactively against this iteration's first +> baseline. New measurement target since stage 2: the mutex-guarded +> inbox + eventfd (the plan's deviation β€” lock-free rings arrive only if +> this number says the mutex costs). Chain order: +> **stage 3 β†’ 22 β†’ 31 β†’ 24 β†’ 23**. ## Goals @@ -122,15 +133,18 @@ exists to close. - **Brainstorm the spec**, settling the four forks; then a plan whose first task is the harness and the baseline file, because nothing downstream means anything without them. -- **Sequence the whole performance arc around this iteration:** - 1. 9b lands β†’ employee compiles and runs β†’ **22 restart-persistence** and - **22 baseline benchmark** (single-thread, both durable and RAM-only). - 2. **7b** (inferred GC + mark-sweep) β†’ re-run 22, record the delta (does - tracing change the write path's tail latency?). - 3. **8** (shard-actor, thread-per-core) β†’ re-run 22 at the connection/ - concurrency scale it unlocks, record the delta. - 4. **23** (io_uring group-commit) β†’ re-run 22's durable write number, record - the delta against the fsync-per-commit baseline β€” the payoff. +- **Sequence the performance chain around this iteration** (rewritten + 2026-08-21 β€” the first version predated 7b and the arc landing first): + 1. Already landed unmeasured: 9b, **7b** (inferred GC + mark-sweep, + 2026-08-18), the arc's **stages 1+2** (fibers + shards, 2026-08-20). + Their deltas are owed retroactively against the first baseline. + 2. Arc **stage 3** (transparent DB actor) lands β†’ **22 runs**: + restart-persistence proof + baseline benchmark, durable and + RAM-only, single- AND multi-shard, plus the mutex-inbox number. + 3. **31** (actor lifecycle), then **24** (chat) β†’ re-run the + concurrency-facing numbers at the connection scale chat unlocks. + 4. **23** (io_uring group-commit) β†’ re-run 22's durable write number, + record the delta against the fsync-per-commit baseline β€” the payoff. - The benchmark harness and its baseline live under `bench/` (or the existing `runtime/bench/`), and `just` gets a `db-bench` recipe kept off the fast path, exactly like `log-watcher::soak`. diff --git a/docs/stories/language-runtime-database/refine/23-io-uring-commit.md b/docs/stories/language-runtime-database/refine/23-io-uring-commit.md index 53ba52f..5cfcf43 100644 --- a/docs/stories/language-runtime-database/refine/23-io-uring-commit.md +++ b/docs/stories/language-runtime-database/refine/23-io-uring-commit.md @@ -22,8 +22,12 @@ > accumulated; (4) startup auto-probe + an env override so CI proves both > paths on one kernel β€” AMENDED: the override is the arc-wide > `WO_IO=uring|epoll` (the arc's T4 owns the probe and the per-shard -> ring; `WO_WAL_MODE` is subsumed). Position: AFTER the 8+11 arc (order -> settled 2026-08-20: 9e β†’ 8+11 β†’ 9f). AMENDED 2026-08-20 (io_uring-first +> ring; `WO_WAL_MODE` is subsumed). Position β€” RE-SEQUENCED 2026-08-21: +> LAST in the concurrency chain, **stage 3 β†’ 22 β†’ 31 β†’ 24 β†’ 23** +> (supersedes the 2026-08-20 old-id ordering "9e β†’ 8+11 β†’ 9f"); the +> per-shard ring already exists (arc T4 landed 2026-08-20, +> `WO_IO=uring|epoll`) β€” this iteration adds the WAL's WRITE+FSYNC +> chains to it. AMENDED 2026-08-20 (io_uring-first > directive): the WAL's WRITE+FSYNC chains ride the SAME per-shard ring > T4 creates for fiber parking β€” one event loop per shard, readiness ops > and durability ops together, exactly the linux reference project's diff --git a/docs/stories/language-runtime-database/refine/24-chat-websocket-workload.md b/docs/stories/language-runtime-database/refine/24-chat-websocket-workload.md index aa0cdbc..5a75d5c 100644 --- a/docs/stories/language-runtime-database/refine/24-chat-websocket-workload.md +++ b/docs/stories/language-runtime-database/refine/24-chat-websocket-workload.md @@ -6,6 +6,12 @@ > **Inserted 2026-08-20** (concurrency-chain refinement): the 8+11 arc's > driving workload, the role log-watcher played for iterations 3–7. Needs > its spec AFTER the arc's β€” it lands at the arc's end and proves it. +> +> **RE-SEQUENCED 2026-08-21**: fourth in the chain, +> **stage 3 β†’ 22 β†’ 31 β†’ 24 β†’ 23** β€” chat cannot be written honestly +> before [iteration 31](31-actor-lifecycle.md) (request/response, +> bounded mailboxes, actor death, timers). Iteration 19 LANDED +> 2026-08-20, so Bytes is available for frame parse/serialize. ## Why this iteration exists @@ -54,11 +60,15 @@ proxy β€” the proxy story extends to WS pass-through, documented). ## Info -- Dependencies: the 8+11 arc (fibers + cross-shard send), the crypto - builtins fork (SHA-1 for the upgrade handshake β€” note: the ledger's - crypto slice lists SHA-256/512; the WS handshake specifically needs - SHA-1, so the builtin set must include it), and the framework's parse - seam (upgrade is an HTTP request until it isn't). +- Dependencies: the 8+11 arc (fibers + cross-shard send β€” stages 1+2 + landed 2026-08-20, stage 3 pending), [iteration 31](31-actor-lifecycle.md) + (request/response, backpressure, death, timers β€” the mechanisms rooms + and presence are made of), iteration 19 (LANDED 2026-08-20 β€” Bytes + carries the frames), the crypto builtins fork (SHA-1 for the upgrade + handshake β€” note: the ledger's crypto slice lists SHA-256/512; the WS + handshake specifically needs SHA-1, so the builtin set must include + it), and the framework's parse seam (upgrade is an HTTP request until + it isn't). - Unparks on landing: the framework ledger's WebSocket/pub-sub rows and the iteration-18 rejection note ("pub/sub REJECTED until 8/11"). diff --git a/docs/stories/language-runtime-database/refine/31-actor-lifecycle.md b/docs/stories/language-runtime-database/refine/31-actor-lifecycle.md new file mode 100644 index 0000000..8a8fd15 --- /dev/null +++ b/docs/stories/language-runtime-database/refine/31-actor-lifecycle.md @@ -0,0 +1,106 @@ +# Iteration 31 β€” actor lifecycle: request/response, backpressure, death, timers + +> Format: fiberloom `product/story-iteration-template`. Part of +> [Story β€” one language, one runtime, one database, one binary](../00-story.md). +> +> **Inserted 2026-08-21** (concurrency-chain re-sequence; the iteration +> was named as "new 31" in the 2026-08-20 code-review re-sequence β€” this +> is its story file). Third in the chain, +> **stage 3 β†’ 22 β†’ 31 β†’ 24 β†’ 23**: chat +> ([iteration 24](24-chat-websocket-workload.md)) cannot be written +> honestly without these four mechanisms. + +## Why this iteration exists + +The arc's stages 1+2 shipped `spawn`/`send` mechanism without lifecycle: +`send` is one-way and callers `sleep`-poll to await an answer; the +mailbox FIFO grows without bound (a hot sender can exhaust a shard's +memory); an actor that traps dies silently (nobody learns, nothing +restarts, its mailbox rots); and there is no timer surface beyond a +fiber blocking in `time.sleep`. Every real serving program β€” chat first β€” +is made of request/response turns, bounded queues, death notices, and +deadlines. Without this iteration the arc is a demo, not a runtime. + +## Goals + +- **Request/response over one-way sends.** A caller can send and park + until the reply arrives β€” one surface, no `sleep`-polling, no second + concurrency vocabulary. Ownership rules unchanged: the request moves, + the reply moves back. +- **Bounded mailboxes with a stated backpressure policy.** A mailbox has + a cap; what happens at the cap (park the sender vs error) is decided by + the spec, one policy, doctrine-pure β€” no silent unbounded growth + anywhere in the runtime. +- **Actor death is observable.** A trap or normal exit produces a signal + another actor can receive; a fiber-trap already isolates (stage 1) β€” + this makes the fact of death deliverable, so a supervisor CAN be + written in `.wo`. +- **Timers as messages.** A deadline or interval delivers to a mailbox + like any other send, riding the shard's existing io_uring/epoll + timeout plumbing (arc T4) β€” no new event loop. + +## Acceptance Criteria (draft β€” the spec refines) + +- What to achieve? + - **Given** an actor that answers requests, + - **when** a caller awaits the reply, + - **then** the caller's fiber parks (the shard serves other fibers, + TID-verified), resumes with the moved reply, and never busy-waits. +- What to achieve? + - **Given** a mailbox at its cap, + - **when** another send arrives, + - **then** the stated backpressure policy fires deterministically, + memory stays bounded (RSS flat under a hot-sender soak), and no + message is silently dropped. +- What to achieve? + - **Given** an actor that traps mid-message, + - **when** it dies, + - **then** its drop maps run (ASan zero leaks), a death signal + reaches the observer that asked for one, and a supervisor written + in `.wo` can respawn it. +- What to achieve? + - **Given** a timer armed by an actor, + - **when** it fires, + - **then** the actor receives it as an ordinary message on its own + shard, and cancelling before expiry means it never delivers. + +## Out Of Scope + +- Supervision TREES / OTP-scale restart policy β€” a `.wo` library once + death signals exist, not runtime policy. +- Priorities and custom scheduling β€” the reduction budget stays the only + fairness mechanism. +- Distributed (cross-process) supervision β€” no network layer exists. +- Changing the ownership-move rule or the unified address surface β€” + iteration 8's decisions stand. + +## Info + +Forks the spec must settle: + +1. **The request/response surface.** A reply-address baked into the + message shape vs a runtime-level call that parks β€” and what the + compiler checks (does a request type name its reply type?). +2. **The backpressure policy at the cap.** Park the sender (natural with + fibers, risks deadlock cycles) vs fail the send (explicit, pushes + handling to the program). One policy, stated; not configurable per + mailbox in v1. +3. **The death-signal shape.** Erlang's link (bidirectional, dies + together) vs monitor (one-way notice) β€” likely monitor-only v1. +4. **The timer surface.** Builtin (`time.after` delivering a message) vs + actor-spawned sleeper fiber β€” and cancellation semantics. + +Sources: the 2026-08-20 code-review findings (the gaps this iteration +answers), the arc plan's stage-2 deviations +([`2026-08-20-shard-fiber-arc.md`](../../../superpowers/plans/2026-08-20-shard-fiber-arc.md) +β€” the unbounded FIFO is deviation 4's mutex inbox), and BEAM precedent +already surveyed in +[`docs/plan/exploration/fibers/00-fibers.md`](../../../plan/exploration/fibers/00-fibers.md). + +## Proposed Solution + +Brainstorm β†’ spec β†’ plan after the arc's stage 3 lands and iteration 22 +has its baseline (the mailbox-cap and inbox-ring decisions want 22's +mutex number). The spec is written against iteration 24's needs β€” chat +names the lifecycle mechanisms it consumes, this iteration names chat as +its first honest consumer.