From 80046556af4f29045ad5b9163dcd25e5a07a62f1 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Mon, 10 Aug 2026 14:11:43 +0200 Subject: [PATCH] docs: update story iterations (pre-compiler-front) - 00: story framing - 01: principles doc - 02: VM core - 04: single binary e2e - 05: language surface - 08: shard-actor runtime - 09: database engine - 10: HTTP service --- .../language-runtime-database/00-story.md | 60 +++++++++++------ .../01-principles-doc.md | 13 ++-- .../language-runtime-database/02-vm-core.md | 32 ++++++--- .../04-single-binary-e2e.md | 4 +- .../05-language-surface.md | 2 +- .../08-shard-actor-runtime.md | 18 ++++- .../09-database-engine.md | 11 +++- .../10-http-service.md | 4 +- .../11-blue-green-deploy.md | 65 ------------------- 9 files changed, 102 insertions(+), 107 deletions(-) delete mode 100644 docs/stories/language-runtime-database/11-blue-green-deploy.md diff --git a/docs/stories/language-runtime-database/00-story.md b/docs/stories/language-runtime-database/00-story.md index 51182df..323e700 100644 --- a/docs/stories/language-runtime-database/00-story.md +++ b/docs/stories/language-runtime-database/00-story.md @@ -5,7 +5,7 @@ **AS** a developer building and operating my own products end to end -**I WANT** a new programming language — with arithmetic, ownership-based memory safety, and garbage collection where I opt in — whose compiler, runtime, and database ship as a single never-stopping Linux binary that can update its own code in place +**I WANT** a new statically-typed systems programming language — object-oriented by default, with ownership-based memory safety (borrow semantics) and per-class `@gc` garbage collection where I opt in — whose compiler, runtime, and database ship as a single never-stopping Linux binary that can update its own code in place **TO** write an application once and run it forever: no external stack to assemble, no database server to operate, and deployments that swap code inside the running process with instant rollback. @@ -17,6 +17,15 @@ - Memory safety without a GC tax: Rust-shaped borrowing (single owner, second-class borrows) checked mostly at compile time, with per-class `@gc` opt-in collected per shard — no global pause exists by construction. +- Static typing all the way down: every slot's type is known at compile + time, so the VM runs untagged 64-bit registers — no `Dynamic`, no boxing + tax, no hashed field lookups (the Haxe→C++ reference workload pays all + three). +- The database is the object model: a `@table` class is a table, a + `ref`/`multi` field is a relation — the `@`-annotations are the built-in + ORM, resolved at compile time. No external mapping layer, and no + user-defined macros (the verdict table's rejection stands; annotations + are compiler-known). - Updates are blue-green **inside** the runtime: propose, approve, compile in-process, atomic switch, previous version resident for instant rollback. - The runtime is a recipe box: once language + runtime + database exist, a @@ -25,9 +34,10 @@ ## Background & Constraints -writeonce today is a declarative Rust-based runtime (Stage 2). This story -evolves it into an object-oriented language (`woc` OCaml compiler, `wovm` C -VM) per the approved specs: C as the runtime's basis (libc only), no +writeonce began as a declarative Rust-based runtime (Stage 2); that chapter +closes here. This story pivots it into a statically-typed, object-oriented +**systems programming language** with its own runtime and database (`woc` +OCaml compiler, `wovm` C VM) per the approved specs: C as the runtime's basis (libc only), no inheritance ever, mutable value semantics for borrowing, shard-per-core concurrency with ownership-moving messages, RAM-authoritative data under a WAL, and the blue-green VM pair for in-runtime deployment. Target OS is @@ -36,21 +46,33 @@ parity. Constraints: OCaml stdlib only, C libc only; docs live under `docs/`; prose-only planning artifacts (no implementation code in stories or iterations); no commits by agents — drafts go to `.dev/commit.md`. -## Iterations (review in this order) +## Iterations (review in this order — the order is chronological) -| # | Iteration | Delivers | -| --- | --- | --- | -| 1 | [Principles doc](01-principles-doc.md) | `docs/00-principles.md` — the doctrine page every later slice links back to | -| 2 | [VM core](02-vm-core.md) | `wovm`: `.wob` loader, register interpreter, arena, borrow word, `@gc` collector | -| 3 | [Compiler front](03-compiler-front.md) | `woc`: lexer → parser → typechecker → ownership pass, diagnostics | -| 4 | [Single binary end-to-end](04-single-binary-e2e.md) | emitter + conformance corpus + `woc build` self-contained binary | -| 5 | [Language surface](05-language-surface.md) | Haxe-parity adoptions: switch, records, optionals, try/catch, statics, modules… | -| 6 | [Program mode + stdlib](06-program-mode-stdlib.md) | `fn main`, exit codes, `fs`/`proc`/`net`/`time`/`json` builtins | -| 7 | [log-watcher proof](07-logwatcher-proof.md) | the driving workload compiled and detecting silent deaths live | -| 8 | [Shard-actor runtime](08-shard-actor-runtime.md) | thread-per-core shards, per-shard heaps, ownership-move messaging | -| 9 | [Database engine](09-database-engine.md) | class-shaped tables, typed WAL + recovery, `insert`/`select` execute | -| 10 | [HTTP service layer](10-http-service.md) | `service` blocks route to VM methods; REST parity with Stage 2 | -| 11 | [Blue-green deploy](11-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback | +| # | Status | Iteration | Delivers | +| --- | --- | --- | --- | +| 1 | ✅ | [Principles doc](01-principles-doc.md) | `docs/00-principles.md` — the doctrine page every later slice links back to | +| 2 | 🔄 | [VM core](02-vm-core.md) | `wovm`: `.wob` loader, register interpreter, arena, borrow word, `@gc` RC (cycles staged to 8) | +| 3 | 🔄 | [Compiler front](03-compiler-front.md) | `woc`: lexer → parser → typechecker → ownership pass, diagnostics | +| 4 | ⬜ | [Single binary end-to-end](04-single-binary-e2e.md) | emitter + conformance corpus + `woc build` self-contained binary | +| 5 | ⬜ | [Language surface](05-language-surface.md) | Haxe-parity adoptions: switch, records, optionals, try/catch, statics, modules… | +| 6 | ⬜ | [Program mode + stdlib](06-program-mode-stdlib.md) | `fn main`, exit codes, `fs`/`proc`/`net`/`time`/`json` builtins | +| 7 | ⬜ | [log-watcher proof](07-logwatcher-proof.md) | the driving workload compiled and detecting silent deaths live | +| 8 | ⬜ | [Shard-actor runtime](08-shard-actor-runtime.md) | thread-per-core shards, per-shard heaps, ownership-move messaging, `@gc` cycle collector | +| 9 | ⬜ | [Database engine](09-database-engine.md) | class-shaped tables, typed WAL + recovery, `insert`/`select` execute | +| 10 | ⬜ | [HTTP service layer](10-http-service.md) | `service` blocks route to VM methods; REST parity with Stage 2 | +| 11 | ⬜ | [Fibers](11-fibers.md) | green threads on the shard scheduler: reduction-budget preemption, park-on-I/O builtins, ownership-move sends | +| 12 | ⬜ | [Blue-green deploy](12-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback | + +Chronology notes: 2 and 3 are the one deliberate overlap — the OOP spec's +first sub-project builds both halves concurrently (both in progress: +`runtime/src/` carries the VM's loader/vm/obj/borrow/gc modules, +`compiler/` has front-end tasks 1–4 of 8); they meet in 4. Iterations 1–7 +complete the single-shard language; 8–12 scale the runtime (8 is the +substrate for everything after: the DB engine lives in its shards, HTTP +serves from them, fibers refine their scheduler, blue-green drains them). +Fibers (11) land before blue-green so the deploy's drain can unwind parked +fibers as part of its own acceptance; blue-green (12) closes the story — +its plan is authored only after 9–10 ship. Review protocol: the developer reads one iteration, approves or amends; the next starts only after approval. Each iteration is an unsplittable @@ -63,7 +85,7 @@ list, and a pointer to the plan document that already sequences its tasks. - Systems track spec: `docs/superpowers/specs/2026-08-01-systems-track-design.md` - Blue-green spec: `docs/superpowers/specs/2026-08-03-blue-green-vm-design.md` - Sample+principles spec: `docs/superpowers/specs/2026-08-07-logwatcher-sample-and-principles-design.md` -- Plan documents: `docs/superpowers/plans/` (plans 1–10), `compiler/plan/` (2, 3, 8 + architecture) +- Plan documents: `docs/superpowers/plans/` (plans 1–10), `docs/plan/compiler/` (2, 3, 8 + architecture) - Roadmap map: `docs/08-project-structure.md` (build sequence) - Behavioral reference workload: `~/projects/log-watcher` (Haxe daemon) diff --git a/docs/stories/language-runtime-database/01-principles-doc.md b/docs/stories/language-runtime-database/01-principles-doc.md index dce9d75..6c59ba9 100644 --- a/docs/stories/language-runtime-database/01-principles-doc.md +++ b/docs/stories/language-runtime-database/01-principles-doc.md @@ -5,16 +5,18 @@ ## Goals -- The repo gains `docs/00-principles.md`: one page stating the twelve +- The repo gains `docs/00-principles.md`: one page stating the thirteen writeonce principles — the doctrine every later iteration links back to instead of re-arguing. -- The twelve: one binary is the whole system; zero dependencies (kernel +- The thirteen: one binary is the whole system; zero dependencies (kernel primitives only); memory safety without a GC tax (MVS ownership, opt-in `@gc`, per-shard collection); no inheritance ever; thread-per-core shards with ownership moves; the runtime never stops (blue-green slots, embedded source); RAM authoritative + WAL durable; samples force the grammar; Linux is the target; capabilities are typed builtins (no FFI); plain - diagnostics are the product; the runtime is a recipe box. + diagnostics are the product; the runtime is a recipe box; statically + typed all the way to the register (no `Dynamic`, untagged VM slots, + annotations as the compile-time ORM). ## Acceptance Criteria @@ -44,6 +46,7 @@ ## Proposed Solution -- Author the page with the twelve principles in the order listed in the - governing spec; verify every link resolves; add the CLAUDE.md pointer +- Author the page with the thirteen principles in the order listed in the + governing spec (the thirteenth — static typing — added 2026-08-08 by + story amendment); verify every link resolves; add the CLAUDE.md pointer line; record the commit draft in `.dev/commit.md`. diff --git a/docs/stories/language-runtime-database/02-vm-core.md b/docs/stories/language-runtime-database/02-vm-core.md index 0b65f0d..ab71e65 100644 --- a/docs/stories/language-runtime-database/02-vm-core.md +++ b/docs/stories/language-runtime-database/02-vm-core.md @@ -8,7 +8,8 @@ - A C, libc-only virtual machine that loads a `.wob` bytecode module and executes method calls with the language's memory model enforced: owned objects with deterministic drops, runtime borrow checks at residual - sites, and `@gc` classes collected without stop-the-world pauses. + sites, and `@gc` classes reference-counted — **RC only in this + iteration**; the cycle collector is staged to iteration 8 (see Info). - Arithmetic and text operations execute correctly — the language's first observable behavior. @@ -27,21 +28,35 @@ every drop, and ASan/Valgrind report zero leaks on both success and trap paths. - What to achieve? - - **Given** a cyclic `@gc` object graph that becomes garbage, - - **when** the collector's budgeted ticks run, - - **then** the cycle is freed within the configured budget and no pause - exceeds the configured slice. + - **Given** `@gc` objects whose aliases are created and dropped, + - **when** the last reference drops (`rc == 0`), + - **then** the object frees immediately, elided RC pairs stay elided, + and ASan/Valgrind report zero leaks on every acyclic fixture. +- What to achieve? + - **Given** a cyclic `@gc` graph that becomes garbage, + - **when** the corpus runs, + - **then** the leak is *expected and asserted* by a must-leak fixture — + the recorded debt iteration 8's cycle collector retires. ## Out Of Scope - The OCaml compiler (iteration 3) — fixtures here are assembled by the test tool, not compiled. - Threads, shards, mailboxes (iteration 8); any DB or HTTP capability. +- The Bacon–Rajan cycle collector — staged to iteration 8, where the + shard's event loop (its per-tick budget host) first exists. Until then a + cyclic `@gc` graph leaks, documented and fixture-asserted. ## Info - The 16-byte object header reserves a shard id now so iteration 8 needs no relayout. +- Staging rationale (decided 2026-08-08): trial deletion under mutation is + the subtlest piece of this iteration, while every sample to date needs + zero cyclic `@gc` (log-watcher: none; pricing: one acyclic cache). Swift + ships RC-without-cycles at mass scale. The header's `IN_CYCLE_BUF` flag + bit and the possible-cycle buffer hook stay reserved, so iteration 8 + adds the scan without relayout or opcode changes. - Format contract: `docs/plan/oop-vm/00-wob-format.md` twinned with `runtime/src/wob.h`; ~40-op register instruction set, computed-goto dispatch. @@ -49,6 +64,7 @@ ## Proposed Solution - Execute the existing plan: `docs/superpowers/plans/2026-08-01-wob-format-and-vm-core.md` - (16 TDD tasks: arena, object model, borrow word, RC + cycle collector, - test assembler, validating loader, interpreter, drop-map unwinding, - builtins, CLI + `just` gate). + (16 TDD tasks: arena, object model, borrow word, RC, test assembler, + validating loader, interpreter, drop-map unwinding, builtins, CLI + + `just` gate) — with its cycle-collector task deferred: that task moves + to iteration 8's plan, replaced here by the must-leak cycle fixture. diff --git a/docs/stories/language-runtime-database/04-single-binary-e2e.md b/docs/stories/language-runtime-database/04-single-binary-e2e.md index 1307ceb..18fd151 100644 --- a/docs/stories/language-runtime-database/04-single-binary-e2e.md +++ b/docs/stories/language-runtime-database/04-single-binary-e2e.md @@ -35,7 +35,7 @@ ## Out Of Scope - Anything beyond the milestone grammar (iteration 5 grows the surface). -- Hot reload / deployment mechanics (iteration 11 — but the self-exec +- Hot reload / deployment mechanics (iteration 12 — but the self-exec trailer this iteration ships is its foundation). ## Info @@ -48,7 +48,7 @@ ## Proposed Solution -- Execute the existing plan: `compiler/plan/2026-08-01-wob-emit-e2e-single-binary.md` +- Execute the existing plan: `docs/plan/compiler/2026-08-01-wob-emit-e2e-single-binary.md` (emitter with ownership lowering + drop maps + vtables, corpus harness, pricing corpus, ownership/trap corpora, gc pump e2e, `woc build` trailer, `just oop-accept` gate over the five spec success criteria). diff --git a/docs/stories/language-runtime-database/05-language-surface.md b/docs/stories/language-runtime-database/05-language-surface.md index ae395fe..f36e100 100644 --- a/docs/stories/language-runtime-database/05-language-surface.md +++ b/docs/stories/language-runtime-database/05-language-surface.md @@ -48,6 +48,6 @@ ## Proposed Solution -- Execute the existing plan: `compiler/plan/2026-08-01-haxe-parity-language.md` +- Execute the existing plan: `docs/plan/compiler/2026-08-01-haxe-parity-language.md` (nine tasks, each shipping its fixtures and error-catalog entries in the same task). diff --git a/docs/stories/language-runtime-database/08-shard-actor-runtime.md b/docs/stories/language-runtime-database/08-shard-actor-runtime.md index eb9372a..461d42a 100644 --- a/docs/stories/language-runtime-database/08-shard-actor-runtime.md +++ b/docs/stories/language-runtime-database/08-shard-actor-runtime.md @@ -11,6 +11,10 @@ state never exists. - The language grows `spawn` and message send; garbage collection stays per-shard, so no global pause appears at any core count. +- The `@gc` story completes here: the Bacon–Rajan cycle collector (staged + out of iteration 2) lands on the shard's own event loop — the per-tick + budget host it was always specified to run on — retiring iteration 2's + documented cycle leak. ## Acceptance Criteria @@ -29,11 +33,17 @@ - **when** code attempts to send it cross-shard, - **then** the compiler rejects it — aliased references cannot cross heap boundaries. +- What to achieve? + - **Given** a cyclic `@gc` object graph that becomes garbage, + - **when** the shard's budgeted collection ticks run, + - **then** the cycle is freed within the configured budget, no pause + exceeds the configured slice, and iteration 2's must-leak fixture + flips to must-collect. ## Out Of Scope -- Fibers/green threads (recorded in the blue-green vision §3; extends this - scheduler later). +- Fibers/green threads — iteration 11 extends this scheduler; research at + `docs/plan/exploration/fibers/00-fibers.md`. - Cross-shard transactions (the database iteration's 2PC concern, later). ## Info @@ -48,4 +58,6 @@ - Execute the existing plan: `docs/superpowers/plans/2026-08-01-shard-actor-vm-runtime.md` (pinned-worker scheduler, shard-stamped heaps, MPSC mailbox rings + mail eventfds, send-as-move with home-routed frees, gc pacing per tick, - spawn/send surface, actor corpus). + spawn/send surface, actor corpus) — plus the cycle-collector task + adopted from iteration 2's plan: possible-cycle buffer on RC decrement, + trial-deletion scan under the per-tick budget. diff --git a/docs/stories/language-runtime-database/09-database-engine.md b/docs/stories/language-runtime-database/09-database-engine.md index 8e0cae6..6b1dee1 100644 --- a/docs/stories/language-runtime-database/09-database-engine.md +++ b/docs/stories/language-runtime-database/09-database-engine.md @@ -6,8 +6,10 @@ ## Goals - The language's oldest promise executes on the new runtime: every class is - a table. `insert` and `select` stop trapping (`DB_STUB` retires) and run - against class-shaped row storage inside the VM's shards. + a table, every `ref`/`multi` field a relation — the `@`-annotations + (`@table`, `@unique`) are the built-in ORM, mapped at compile time with no + external layer. `insert` and `select` stop trapping (`DB_STUB` retires) + and run against class-shaped row storage inside the VM's shards. - Data survives anything: a typed write-ahead log with ack-after-fsync, parallel boot replay, and a crash battery proving no committed row is ever lost and no half-applied transaction ever visible. @@ -45,6 +47,11 @@ - Doctrine: RAM is authoritative; the WAL makes it durable; indexes drift unless writes go through the row API — the Rust runtime learned this lesson, the C engine enforces it. +- Annotations, not macros: `@table`/`@unique` are compiler-known and + resolved at compile time; user-defined macros stay rejected (systems-track + verdict table). A `ref T` field compiles to a typed row id (FK); `multi T` + to the owning-side collection edge — the relation model the examples + (pricing's `Product`/`Price`) already write. - The wo-db overlap manifest keeps the C++ prototype and this engine answer-compatible where features overlap. diff --git a/docs/stories/language-runtime-database/10-http-service.md b/docs/stories/language-runtime-database/10-http-service.md index 2a3f085..a118054 100644 --- a/docs/stories/language-runtime-database/10-http-service.md +++ b/docs/stories/language-runtime-database/10-http-service.md @@ -16,7 +16,7 @@ - What to achieve? - **Given** the blog sample's `service` declarations, - - **when** the compiled binary boots and `reference/rest/blog.rest` + - **when** the compiled binary boots and `.dev/reference/rest/blog.rest` runs against it, - **then** every request in the smoke file answers as documented — including the intentional 501/405/404 responses. @@ -36,7 +36,7 @@ ## Out Of Scope - WebSocket/live subscriptions and UI (the parked `##ui` story). -- The management plane endpoints (iteration 11 builds them on this +- The management plane endpoints (iteration 12 builds them on this machinery). ## Info diff --git a/docs/stories/language-runtime-database/11-blue-green-deploy.md b/docs/stories/language-runtime-database/11-blue-green-deploy.md deleted file mode 100644 index 74b7e93..0000000 --- a/docs/stories/language-runtime-database/11-blue-green-deploy.md +++ /dev/null @@ -1,65 +0,0 @@ -# Iteration 11 — blue-green in-runtime deployment - -> Format: `product/story-iteration-template`. Part of -> [Story — one language, one runtime, one database, one binary](00-story.md). - -## Goals - -- The story's closing promise: the running binary updates its own code. - Two fixed VM slots (activity alternating); a proposal pipeline — - propose → approve → in-runtime compile → additive schema migration → - load → health → atomic switch — with the previous version staying - resident as the instant rollback target. -- The developer drives it remotely: `wo remote pull / propose / diff / - approve / rollback / status` over a loopback management transport, - every stage streaming live and WAL-audited. - -## Acceptance Criteria - -- What to achieve? - - **Given** a running fixture app and an additive code+schema change, - - **when** the developer proposes and approves it, - - **then** the deploy completes with zero dropped requests (in-flight - work drains on the old slot), and the binary's embedded source - trailer matches the new active version afterward. -- What to achieve? - - **Given** a failure at any pipeline stage (compile diagnostic, - destructive-change rejection, load failure, health failure, drain - timeout), - - **when** it occurs, - - **then** the active slot keeps serving untouched, the failure is - visible in the SSE stream and the WAL trail, and a destructive - change was rejected at propose time naming the offending - declaration. -- What to achieve? - - **Given** a completed deploy, - - **when** `wo remote rollback` runs, - - **then** the previous version serves again in under one second with - no compile and no data change — additive-only migration guarantees - old code runs correctly against the migrated schema. -- What to achieve? - - **Given** kill -9 during COMPILING / MIGRATING / SWITCHING / - trailer-rewrite, - - **when** the unit restarts, - - **then** it serves one consistent version and the WAL shows whole - migrations only. - -## Out Of Scope - -- Script-based/destructive migrations, in-runtime editing workspace, - MCP/agent wrapper over the management plane — all recorded follow-ups. -- Fibers (the vision's §3; a scheduler concern, not a deploy concern). - -## Info - -- Approved spec: `docs/superpowers/specs/2026-08-03-blue-green-vm-design.md`; - its implementation plan is deliberately authored only after iterations - 9–10 ship (prerequisites: a catalog to diff, HTTP machinery to build on). -- VMs own code; the engine owns data — the separation that makes the - switch cheap and rollback unconditional. - -## Proposed Solution - -- Author the implementation plan from the approved spec once iterations - 9–10 land, then execute it (slots, deploy state machine, additive - differ, management surface + SSE, `wo remote` verbs, crash battery).