From 9279175f26c433ad27431149f293a30c3291c4e7 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Mon, 4 May 2026 13:39:58 +0200 Subject: [PATCH] scaffold sibling crates, multi-app ecommerce, REST + concurrency docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - scaffold 14 placeholder crates (app, db, engine, gen, http, logic, policy, ql, service, sub, txn, ui, value, wal) — empty Cargo.toml + src/lib.rs to receive code phase-by-phase from `rt` - restructure docs/examples/ecommerce into multi-app layout: apps/admin and apps/storefront, with shared/ types/logic/components, per-app app.wo + wo.toml, and reusable .htmlx components (layout, money, order-row) - add reference/rest/{blog,ecommerce}.rest — VS Code/JetBrains HTTP request files driving the running prototype, including 501/404/405 expectations for stubbed endpoints - add docs/plan/09-concurrency-scaleout.md and docs/plan/ui/00-overview.md; refine docs/plan/assembly/02-writeonce-stance.md - refresh templates (about, article, header/footer, home, layout, styles) and add static favicon/logo - add infra/sync.sh and tighten .gitignore for reference/ symlinks --- .gitignore | 8 +- crates/app/Cargo.toml | 9 + crates/app/src/lib.rs | 20 ++ crates/db/Cargo.toml | 9 + crates/db/src/lib.rs | 27 ++ crates/engine/Cargo.toml | 9 + crates/engine/src/lib.rs | 19 ++ crates/gen/Cargo.toml | 9 + crates/gen/src/lib.rs | 21 ++ crates/http/Cargo.toml | 9 + crates/http/src/lib.rs | 20 ++ crates/logic/Cargo.toml | 9 + crates/logic/src/lib.rs | 19 ++ crates/policy/Cargo.toml | 9 + crates/policy/src/lib.rs | 25 ++ crates/ql/Cargo.toml | 9 + crates/ql/src/lib.rs | 13 + crates/service/Cargo.toml | 9 + crates/service/src/lib.rs | 19 ++ crates/sub/Cargo.toml | 9 + crates/sub/src/lib.rs | 21 ++ crates/txn/Cargo.toml | 9 + crates/txn/src/lib.rs | 19 ++ crates/ui/Cargo.toml | 9 + crates/ui/src/lib.rs | 24 ++ crates/value/Cargo.toml | 9 + crates/value/src/lib.rs | 14 + crates/wal/Cargo.toml | 9 + crates/wal/src/lib.rs | 17 + docs/examples/ecommerce/README.md | 311 +++++------------- docs/examples/ecommerce/apps/admin/app.wo | 19 ++ .../admin/ui/orders/orders.wo} | 16 +- docs/examples/ecommerce/apps/admin/wo.toml | 24 ++ .../examples/ecommerce/apps/storefront/app.wo | 19 ++ .../storefront/ui/home/home.wo} | 14 +- .../storefront/ui/orders/orders.wo} | 2 +- .../ecommerce/apps/storefront/wo.toml | 26 ++ .../ecommerce/shared/components/layout.htmlx | 42 +++ .../ecommerce/shared/components/money.htmlx | 6 + .../shared/components/order-row.htmlx | 23 ++ .../ecommerce/{ => shared}/logic/checkout.wo | 0 .../{app.wo => shared/logic/seed.wo} | 23 +- .../ecommerce/{ => shared}/types/customer.wo | 0 .../ecommerce/{ => shared}/types/order.wo | 0 .../ecommerce/{ => shared}/types/product.wo | 0 .../ecommerce/{ => shared}/types/purchase.wo | 0 docs/examples/ecommerce/wo.toml | 28 +- docs/plan/09-concurrency-scaleout.md | 125 +++++++ docs/plan/assembly/02-writeonce-stance.md | 5 +- docs/plan/ui/00-overview.md | 213 ++++++++++++ infra/sync.sh | 17 + reference/rest/blog.rest | 181 ++++++++++ reference/rest/ecommerce.rest | 142 ++++++++ static/favicon.svg | 4 + static/logo.svg | 5 + templates/about.htmlx | 21 +- templates/article.htmlx | 15 +- templates/components/article-card.htmlx | 1 + templates/contact.htmlx | 2 - templates/footer.htmlx | 2 +- templates/header.htmlx | 8 +- templates/home.htmlx | 16 +- templates/layout.htmlx | 1 + templates/styles/main.css | 77 ++++- 64 files changed, 1511 insertions(+), 289 deletions(-) create mode 100644 crates/app/Cargo.toml create mode 100644 crates/app/src/lib.rs create mode 100644 crates/db/Cargo.toml create mode 100644 crates/db/src/lib.rs create mode 100644 crates/engine/Cargo.toml create mode 100644 crates/engine/src/lib.rs create mode 100644 crates/gen/Cargo.toml create mode 100644 crates/gen/src/lib.rs create mode 100644 crates/http/Cargo.toml create mode 100644 crates/http/src/lib.rs create mode 100644 crates/logic/Cargo.toml create mode 100644 crates/logic/src/lib.rs create mode 100644 crates/policy/Cargo.toml create mode 100644 crates/policy/src/lib.rs create mode 100644 crates/ql/Cargo.toml create mode 100644 crates/ql/src/lib.rs create mode 100644 crates/service/Cargo.toml create mode 100644 crates/service/src/lib.rs create mode 100644 crates/sub/Cargo.toml create mode 100644 crates/sub/src/lib.rs create mode 100644 crates/txn/Cargo.toml create mode 100644 crates/txn/src/lib.rs create mode 100644 crates/ui/Cargo.toml create mode 100644 crates/ui/src/lib.rs create mode 100644 crates/value/Cargo.toml create mode 100644 crates/value/src/lib.rs create mode 100644 crates/wal/Cargo.toml create mode 100644 crates/wal/src/lib.rs create mode 100644 docs/examples/ecommerce/apps/admin/app.wo rename docs/examples/ecommerce/{ui/admin_orders.wo => apps/admin/ui/orders/orders.wo} (77%) create mode 100644 docs/examples/ecommerce/apps/admin/wo.toml create mode 100644 docs/examples/ecommerce/apps/storefront/app.wo rename docs/examples/ecommerce/{ui/storefront.wo => apps/storefront/ui/home/home.wo} (50%) rename docs/examples/ecommerce/{ui/order_tracker.wo => apps/storefront/ui/orders/orders.wo} (98%) create mode 100644 docs/examples/ecommerce/apps/storefront/wo.toml create mode 100644 docs/examples/ecommerce/shared/components/layout.htmlx create mode 100644 docs/examples/ecommerce/shared/components/money.htmlx create mode 100644 docs/examples/ecommerce/shared/components/order-row.htmlx rename docs/examples/ecommerce/{ => shared}/logic/checkout.wo (100%) rename docs/examples/ecommerce/{app.wo => shared/logic/seed.wo} (69%) rename docs/examples/ecommerce/{ => shared}/types/customer.wo (100%) rename docs/examples/ecommerce/{ => shared}/types/order.wo (100%) rename docs/examples/ecommerce/{ => shared}/types/product.wo (100%) rename docs/examples/ecommerce/{ => shared}/types/purchase.wo (100%) create mode 100644 docs/plan/09-concurrency-scaleout.md create mode 100644 docs/plan/ui/00-overview.md create mode 100755 infra/sync.sh create mode 100644 reference/rest/blog.rest create mode 100644 reference/rest/ecommerce.rest create mode 100644 static/favicon.svg create mode 100644 static/logo.svg delete mode 100644 templates/contact.htmlx diff --git a/.gitignore b/.gitignore index b9a4c85..3c653f8 100644 --- a/.gitignore +++ b/.gitignore @@ -13,10 +13,12 @@ # Legacy blog content and data (v1 writeonce storage) /content -# Symlink to the Linux kernel source tree for research — user-specific -# absolute path; each contributor sets their own via +# Symlinks to research source trees — user-specific absolute paths. +# Each contributor sets their own via: # ln -s reference/linux +# ln -s reference/go /reference/linux +/reference/go # Editor / OS noise — left broad on purpose so a contributor doesn't # accidentally commit their IDE scratch or macOS metadata. @@ -27,3 +29,5 @@ /.vscode/* !/.vscode/settings.json.example !/.vscode/extensions.json + +/prototypes \ No newline at end of file diff --git a/crates/app/Cargo.toml b/crates/app/Cargo.toml new file mode 100644 index 0000000..b389257 --- /dev/null +++ b/crates/app/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "app" +version = "0.1.0" +edition = "2021" +description = "writeonce app manifest — `##app` routes, startup hooks, theme, i18n" + +[lib] +name = "app" +path = "src/lib.rs" diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs new file mode 100644 index 0000000..ddf0408 --- /dev/null +++ b/crates/app/src/lib.rs @@ -0,0 +1,20 @@ +//! `app` — `##app` manifest: routes, theme, i18n, startup hooks. +//! +//! **Status: placeholder.** Phase 6 (see +//! [06-lowcode-fullstack.md § `##app`]( +//! ../../../docs/runtime/database/06-lowcode-fullstack.md)). +//! +//! One `##app` block per project. Declares: +//! * the static route table (URL path → `ui.` binding, possibly +//! parameterised by dynamic segments like `/article/:slug`) +//! * cross-entity policies ("Admin bypasses row-level filters on every type") +//! * `on startup do …` hooks — idempotent seed code that runs once before +//! [`http`](../http/index.html) binds a listening socket +//! * theme tokens, i18n locale set, project metadata +//! +//! Consumed by [`ui`](../ui/index.html) for route rendering and by +//! [`policy`](../policy/index.html) for the cross-entity rules block. +//! +//! The [`docs/examples/blog/app.wo`](../../../docs/examples/blog/app.wo) and +//! [`docs/examples/ecommerce/app.wo`](../../../docs/examples/ecommerce/app.wo) +//! files are the reference shapes. diff --git a/crates/db/Cargo.toml b/crates/db/Cargo.toml new file mode 100644 index 0000000..4ba468e --- /dev/null +++ b/crates/db/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "db" +version = "0.1.0" +edition = "2021" +description = "writeonce top-level database facade — `open()`, `Tx`, `Query`, `Subscribe` over the engine" + +[lib] +name = "db" +path = "src/lib.rs" diff --git a/crates/db/src/lib.rs b/crates/db/src/lib.rs new file mode 100644 index 0000000..84f9362 --- /dev/null +++ b/crates/db/src/lib.rs @@ -0,0 +1,27 @@ +//! `wo-db` — the Rust-facing database SDK. +//! +//! **Status: placeholder.** Phases 2–4 integrator. +//! +//! The top-level crate that the rest of the workspace (`wo-rt`, `wo-http`, +//! future `wo-serve` integration glue) depends on. Composes: +//! +//! | | | +//! | --- | --- | +//! | [`ql`](../ql/index.html) | parse | +//! | [`value`](../value/index.html) | row representation | +//! | [`engine`](../engine/index.html) | storage + execution | +//! | [`txn`](../txn/index.html) | transaction coordinator | +//! | [`wal`](../wal/index.html) | durability (Phase 3) | +//! | [`sub`](../sub/index.html) | live subscriptions (Phase 4) | +//! +//! Public surface the rest of the workspace will call: +//! +//! ```text +//! let db = db::open(&catalog, &options)?; +//! db.wo(ctx, ".wo source", params).await?; // raw DML +//! db.tx(ctx, |tx| async move { ... }).await?; +//! let sub = db.subscribe(ctx, "LIVE SELECT ...", params).await?; +//! ``` +//! +//! See [05-go-sdk.md](../../../docs/runtime/database/05-go-sdk.md) — the Go +//! SDK's shape mirrors this one; both speak the same `.wo` wire protocol. diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml new file mode 100644 index 0000000..5a344a6 --- /dev/null +++ b/crates/engine/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "engine" +version = "0.1.0" +edition = "2021" +description = "writeonce in-memory executor — relational, document, and graph storage over a shared catalog" + +[lib] +name = "engine" +path = "src/lib.rs" diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs new file mode 100644 index 0000000..7e1f65d --- /dev/null +++ b/crates/engine/src/lib.rs @@ -0,0 +1,19 @@ +//! `wo-engine` — the in-memory executor. +//! +//! **Status: placeholder.** Phase 2 (see +//! [02-wo-language.md](../../../docs/runtime/database/02-wo-language.md)) +//! designs the three storage paradigms; Phase 3 +//! ([03-inmemory-engine.md](../../../docs/runtime/database/03-inmemory-engine.md)) +//! details the RAM-primary layout, arenas, MVCC version chains. +//! +//! The engine owns: +//! * relational tables — per-type `BTreeMap` today, B+ tree when +//! sized appropriately +//! * document collections — shape-free `Value::Object` rows; LSM when Phase 3 +//! activates +//! * graph nodes + edges — adjacency over the type catalog +//! * a schema catalog compiled from `wo-ql` output +//! +//! Stage 2 ships this as a module inside [`rt`](../rt/index.html). +//! It extracts here when `wo-wal` and `wo-txn` need to reach into the same +//! structures. diff --git a/crates/gen/Cargo.toml b/crates/gen/Cargo.toml new file mode 100644 index 0000000..f4a9c89 --- /dev/null +++ b/crates/gen/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "gen" +version = "0.1.0" +edition = "2021" +description = "writeonce codegen — `.wo` schema → typed Go / TypeScript / Rust / Python clients" + +[lib] +name = "gen" +path = "src/lib.rs" diff --git a/crates/gen/src/lib.rs b/crates/gen/src/lib.rs new file mode 100644 index 0000000..eb43530 --- /dev/null +++ b/crates/gen/src/lib.rs @@ -0,0 +1,21 @@ +//! `wo-gen` — client-SDK codegen. +//! +//! **Status: placeholder.** Phase 5 (see +//! [05-go-sdk.md § Typed SDK via `.wo` Schema Codegen]( +//! ../../../docs/runtime/database/05-go-sdk.md)). +//! +//! Reads a compiled catalog (the same output [`ql`](../ql/index.html) + +//! [`engine`](../engine/index.html) produce from a `.wo` source tree) +//! and emits typed client code: +//! +//! * **Go** — structs with `wo:"column"` tags, `*Client`, +//! `TypedSubscription[T]` generics over channels +//! * **TypeScript** — types + `fetch` + WebSocket subscriptions +//! * **Rust** — `#[derive(Deserialize)]` structs + `impl Stream` +//! * **Python** — dataclasses + `async for delta in sub` +//! * **OpenAPI / GraphQL SDL** — machine-generated contracts served by +//! [`http`](../http/index.html) +//! +//! Invoked as `wo gen sdk --lang go --out ./client` via the toolchain; a bin +//! target will be added when the crate has real behaviour. Until then this is +//! an empty library scaffold. diff --git a/crates/http/Cargo.toml b/crates/http/Cargo.toml new file mode 100644 index 0000000..5f4d53b --- /dev/null +++ b/crates/http/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "http" +version = "0.1.0" +edition = "2021" +description = "writeonce wire protocol — native binary + GraphQL over WebSocket + REST dispatch" + +[lib] +name = "http" +path = "src/lib.rs" diff --git a/crates/http/src/lib.rs b/crates/http/src/lib.rs new file mode 100644 index 0000000..77178f3 --- /dev/null +++ b/crates/http/src/lib.rs @@ -0,0 +1,20 @@ +//! `wo-http` — HTTP + WebSocket + native wire protocol for the `.wo` runtime. +//! +//! **Status: placeholder.** Phase 4 (see +//! [04-client-api.md](../../../docs/runtime/database/04-client-api.md)). +//! +//! Three protocol surfaces, one dispatch: +//! * **REST** — JSON in/out, one route per `service rest` operation. +//! * **GraphQL over WebSocket** — SDL auto-generated from the schema; query, +//! mutation, `subscription` all routed through the same planner. +//! * **Native binary** — typed wire codec for `wo-db` and the Go SDK. +//! +//! Authentication middleware (`WithAPIKey`, `WithJWT`, `WithMTLS`) lives in +//! this crate — cross-cutting over every protocol surface. +//! +//! Stage 2 ships a minimal axum-based REST server inside +//! [`rt::server`](../rt/server/index.html). It migrates here when +//! Phase 4 activates and GraphQL / native surfaces join REST. +//! +//! Not the v1 `reference/crates/wo-http/` — that was a from-scratch HTTP +//! parser for the old single-threaded event loop. diff --git a/crates/logic/Cargo.toml b/crates/logic/Cargo.toml new file mode 100644 index 0000000..cb4d7d0 --- /dev/null +++ b/crates/logic/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "logic" +version = "0.1.0" +edition = "2021" +description = "writeonce trigger + stored-fn compiler — type-attached `on ` and cross-entity `##logic`" + +[lib] +name = "logic" +path = "src/lib.rs" diff --git a/crates/logic/src/lib.rs b/crates/logic/src/lib.rs new file mode 100644 index 0000000..be5a7c6 --- /dev/null +++ b/crates/logic/src/lib.rs @@ -0,0 +1,19 @@ +//! `wo-logic` — triggers and stored-function interpreter. +//! +//! **Status: placeholder.** Phase 6 (see +//! [06-lowcode-fullstack.md § `##logic`]( +//! ../../../docs/runtime/database/06-lowcode-fullstack.md)). +//! +//! Two sources of runtime code: +//! * **Type-attached triggers** — `on insert | update | delete [when ] +//! do ` declared inside a `type` body. Fires inside the committing +//! transaction so timestamp stamps (`paid_at`, `shipped_at`) are atomic +//! with the state change, never observable half-way. +//! * **Standalone `##logic`** — cross-entity workflows like the ecommerce +//! on-order-placed hook that decrements inventory on every line item. +//! * **`fn name(args) in txn ... { ... }`** — transactional stored +//! functions (the `checkout` in `docs/examples/ecommerce/logic/`). +//! +//! All three compile to the same intermediate form that [`txn`](../txn/index.html) +//! invokes during commit (for triggers) or directly on the main loop (for +//! `fn` calls from REST/native wire). diff --git a/crates/policy/Cargo.toml b/crates/policy/Cargo.toml new file mode 100644 index 0000000..b8f8fc6 --- /dev/null +++ b/crates/policy/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "policy" +version = "0.1.0" +edition = "2021" +description = "writeonce policy engine — RBAC + row-level rules compiled into planner rewrites" + +[lib] +name = "policy" +path = "src/lib.rs" diff --git a/crates/policy/src/lib.rs b/crates/policy/src/lib.rs new file mode 100644 index 0000000..7bb3f08 --- /dev/null +++ b/crates/policy/src/lib.rs @@ -0,0 +1,25 @@ +//! `wo-policy` — RBAC + row-level policies. +//! +//! **Status: placeholder.** Phase 6 (see +//! [06-lowcode-fullstack.md](../../../docs/runtime/database/06-lowcode-fullstack.md)). +//! +//! Compiles type-attached `policy` blocks (and standalone `##policy` blocks +//! for cross-entity rules) into predicates that [`engine`](../engine/index.html) +//! AND-s into every query at plan time. The rule is enforced once, at the +//! planner, so no caller can bypass it — not the REST handler, not the +//! `wo-db` SDK, not an admin script calling `fn` directly. +//! +//! Shape of a compiled policy: +//! +//! ```text +//! for type Article: +//! read: published == true OR $session.role == Admin +//! OR author == $session.user +//! write: $session.role == Admin OR ($session.role == Author && author == $session.user) +//! delete: $session.role == Admin +//! ``` +//! +//! The planner merges the read predicate as a `WHERE` conjunct, the write +//! predicate as an assertion inside UPDATE/INSERT codegen, and delete as a +//! DELETE guard. Same mechanism as Postgres RLS — just authored declaratively +//! in the `type` block instead of via SQL migrations. diff --git a/crates/ql/Cargo.toml b/crates/ql/Cargo.toml new file mode 100644 index 0000000..96d7bcc --- /dev/null +++ b/crates/ql/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "ql" +version = "0.1.0" +edition = "2021" +description = "writeonce query language — lexer, parser, AST for the `.wo` grammar" + +[lib] +name = "ql" +path = "src/lib.rs" diff --git a/crates/ql/src/lib.rs b/crates/ql/src/lib.rs new file mode 100644 index 0000000..f387c3d --- /dev/null +++ b/crates/ql/src/lib.rs @@ -0,0 +1,13 @@ +//! `wo-ql` — the `.wo` grammar: lexer, parser, AST. +//! +//! **Status: placeholder.** Phase 2 of the [`.wo` runtime design]( +//! ../../../docs/runtime/database/02-wo-language.md). +//! +//! Stage 2 of the runtime (shipped) carries the lexer/parser/AST inside +//! [`rt`](../rt/index.html) as part of its monolithic cut. This crate +//! is where those modules move to when Stage 3+ needs the grammar from +//! multiple places — the HTTP raw-query handler, `wo-gen`, the server-side +//! `fn` interpreter. +//! +//! Reference implementation: the C++ prototype at +//! [`prototypes/wo-db/`](../../../prototypes/wo-db/). diff --git a/crates/service/Cargo.toml b/crates/service/Cargo.toml new file mode 100644 index 0000000..e1741da --- /dev/null +++ b/crates/service/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "service" +version = "0.1.0" +edition = "2021" +description = "writeonce service dispatcher — type-attached `service rest/graphql/native` and standalone `##service` bundles" + +[lib] +name = "service" +path = "src/lib.rs" diff --git a/crates/service/src/lib.rs b/crates/service/src/lib.rs new file mode 100644 index 0000000..b7a0c1f --- /dev/null +++ b/crates/service/src/lib.rs @@ -0,0 +1,19 @@ +//! `service` — endpoint registration and dispatch. +//! +//! **Status: placeholder.** Phase 6 (see +//! [06-lowcode-fullstack.md § `##service`]( +//! ../../../docs/runtime/database/06-lowcode-fullstack.md)). +//! +//! Walks every `service rest "/api/..." expose ...` block declared on a type +//! (and every standalone `##service` bundle for multi-entity APIs) and binds +//! handlers on [`http`](../http/index.html)'s router. Each exposed operation +//! (`list` / `get` / `create` / `update` / `delete` / `subscribe` / `me`) +//! maps to a predetermined shape — declarative CRUD is the whole point. +//! +//! `fn` endpoints (`POST /api/fn/checkout`) are also dispatched from here — +//! the service declares the function-as-endpoint wiring, [`logic`](../logic/index.html) +//! executes. +//! +//! Stage 2 does this work inline inside [`rt::server`](../rt/server/index.html). +//! It extracts here when Phase 6 adds GraphQL + native-protocol surfaces that +//! share the same service-block source of truth. diff --git a/crates/sub/Cargo.toml b/crates/sub/Cargo.toml new file mode 100644 index 0000000..daacd43 --- /dev/null +++ b/crates/sub/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "sub" +version = "0.1.0" +edition = "2021" +description = "writeonce live subscriptions — LIVE query registry + delta frames on commit" + +[lib] +name = "sub" +path = "src/lib.rs" diff --git a/crates/sub/src/lib.rs b/crates/sub/src/lib.rs new file mode 100644 index 0000000..07f4136 --- /dev/null +++ b/crates/sub/src/lib.rs @@ -0,0 +1,21 @@ +//! `wo-sub` — the live-subscription engine. +//! +//! **Status: placeholder.** Phase 4 (see +//! [04-client-api.md](../../../docs/runtime/database/04-client-api.md)). +//! +//! Responsibilities: +//! * per-connection subscription registry — predicate + bound parameters +//! * commit-path dispatch: each [`txn`](../txn/index.html) commit +//! walks the registry, matches predicates against the delta set, and +//! pushes `Insert` / `Update` / `Delete` frames into per-subscription +//! ring buffers +//! * back-pressure policy (drop-and-resync / coalesce / disconnect) — +//! mirrors the client-side policy in [`wo-db`](../db/index.html) +//! / the Go SDK +//! +//! Stage 2 stubs the `/api//live` endpoint to 501 in +//! [`rt::server`](../rt/server/index.html). Stage 3 wires this crate +//! behind that route and upgrades the connection to WebSocket. +//! +//! Not related to the v1 `reference/crates/wo-sub/` crate — that one did +//! polling-based diff delivery over SSE for the old flat-file blog. diff --git a/crates/txn/Cargo.toml b/crates/txn/Cargo.toml new file mode 100644 index 0000000..86b3e70 --- /dev/null +++ b/crates/txn/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "txn" +version = "0.1.0" +edition = "2021" +description = "writeonce transaction coordinator — MVCC, snapshot isolation, `RETURNING` alias table" + +[lib] +name = "txn" +path = "src/lib.rs" diff --git a/crates/txn/src/lib.rs b/crates/txn/src/lib.rs new file mode 100644 index 0000000..9058a57 --- /dev/null +++ b/crates/txn/src/lib.rs @@ -0,0 +1,19 @@ +//! `wo-txn` — the cross-paradigm transaction coordinator. +//! +//! **Status: placeholder.** Phase 2 milestone 3 (see +//! [02-wo-language.md § Cross-Paradigm Transaction Coordinator]( +//! ../../../docs/runtime/database/02-wo-language.md)). +//! +//! Responsibilities: +//! * `txn_id` allocation + snapshot timestamps +//! * ordering commits across `wo-engine`'s three storage paradigms +//! * the transaction-scoped **`RETURNING` alias table** — the state that +//! lets `INSERT … RETURNING id AS oid` thread `$oid` into a later +//! `CREATE (u)-[:PURCHASED {order_id: $oid}]->(p)` in the same +//! `BEGIN … COMMIT` block +//! * driving the in-process 2PC between the three engines on commit +//! +//! Under the single-threaded event-loop design (see +//! [§ Concurrency Model](../../../docs/runtime/database/02-wo-language.md#concurrency-model)) +//! this reduces to a sequential counter + a per-txn name table. When the +//! runtime grows past one core via sharding, this crate owns cross-shard 2PC. diff --git a/crates/ui/Cargo.toml b/crates/ui/Cargo.toml new file mode 100644 index 0000000..7581634 --- /dev/null +++ b/crates/ui/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "ui" +version = "0.1.0" +edition = "2021" +description = "writeonce UI renderer — `##ui` screens compile to SSR HTML + thin client runtime" + +[lib] +name = "ui" +path = "src/lib.rs" diff --git a/crates/ui/src/lib.rs b/crates/ui/src/lib.rs new file mode 100644 index 0000000..fc589ba --- /dev/null +++ b/crates/ui/src/lib.rs @@ -0,0 +1,24 @@ +//! `ui` — `##ui` screens → SSR HTML + client-side delta runtime. +//! +//! **Status: placeholder.** Phase 6 (see +//! [06-lowcode-fullstack.md § `##ui`]( +//! ../../../docs/runtime/database/06-lowcode-fullstack.md)). +//! +//! Declarative screens compile to: +//! * a **server-rendered HTML** tree — a compact template + data-binding +//! form computed from the screen's `source:` projection and `columns:` / +//! `sections:` declarations +//! * a **client-side runtime** (~50 KB vanilla JS) — opens a WebSocket to +//! the engine's [`sub`](../sub/index.html) endpoint, binds incoming delta +//! frames to DOM fragments by row key, handles sort / filter / paginate +//! without refetching +//! * an **auto-admin fallback** — any declared `type` without an explicit +//! `##ui` gets a generic CRUD screen +//! +//! `live: true` on a screen auto-generates a matching `LIVE SELECT` that the +//! client runtime subscribes to — no hand-written subscription code on the UI +//! side. +//! +//! The [`docs/examples/blog/ui/`](../../../docs/examples/blog/ui/) and +//! [`docs/examples/ecommerce/ui/`](../../../docs/examples/ecommerce/ui/) +//! directories are the reference shapes this crate has to handle. diff --git a/crates/value/Cargo.toml b/crates/value/Cargo.toml new file mode 100644 index 0000000..b8fbb5f --- /dev/null +++ b/crates/value/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "value" +version = "0.1.0" +edition = "2021" +description = "writeonce tagged `Value` + dotted-path utilities" + +[lib] +name = "value" +path = "src/lib.rs" diff --git a/crates/value/src/lib.rs b/crates/value/src/lib.rs new file mode 100644 index 0000000..a2ec3f4 --- /dev/null +++ b/crates/value/src/lib.rs @@ -0,0 +1,14 @@ +//! `wo-value` — the runtime's tagged `Value` type plus dotted-path helpers. +//! +//! **Status: placeholder.** Phase 2 foundation. +//! +//! Carries the union of all scalar + document + array shapes the engines pass +//! around: `Null`, `Bool`, `Int`, `Str`, `Array`, `Object`. Plus `fetch_path` +//! / `assign_path` utilities that drive the dotted-access semantics shared by +//! SQL expressions, document `UPDATE SET meta.x.y = z`, and Cypher projections +//! (see [the two-layer language spec]( +//! ../../../docs/runtime/database/02-wo-language.md)). +//! +//! Stage 2 keeps these inside [`rt`](../rt/index.html)'s `engine` module; +//! they extract here once a second consumer (the WAL serializer, the wire codec) +//! needs them. diff --git a/crates/wal/Cargo.toml b/crates/wal/Cargo.toml new file mode 100644 index 0000000..bdf8ad6 --- /dev/null +++ b/crates/wal/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "wal" +version = "0.1.0" +edition = "2021" +description = "writeonce write-ahead log — io_uring + fsync + crash recovery" + +[lib] +name = "wal" +path = "src/lib.rs" diff --git a/crates/wal/src/lib.rs b/crates/wal/src/lib.rs new file mode 100644 index 0000000..baf8184 --- /dev/null +++ b/crates/wal/src/lib.rs @@ -0,0 +1,17 @@ +//! `wo-wal` — the write-ahead log. +//! +//! **Status: placeholder.** Phase 3 (see +//! [03-inmemory-engine.md](../../../docs/runtime/database/03-inmemory-engine.md)). +//! +//! Responsibilities: +//! * append committed txn records to an on-disk ring buffer via `io_uring` +//! * issue `IORING_OP_FSYNC` with a linked SQE for durability +//! * batch concurrent commits into one fsync per tick (group commit) +//! * drive crash recovery by replaying the log from the last checkpoint +//! +//! Under the single-threaded event-loop design, the WAL is owned directly by +//! the main loop — no separate WAL-writer thread. `io_uring`'s kernel-owned +//! SQPOLL handler does the draining; userland just submits and parks on CQEs. +//! +//! This is the crate that turns the RAM-primary engine from "cache" into +//! "database": commits aren't acked until their WAL record is on the SSD. diff --git a/docs/examples/ecommerce/README.md b/docs/examples/ecommerce/README.md index f2a6932..ae8b3fd 100644 --- a/docs/examples/ecommerce/README.md +++ b/docs/examples/ecommerce/README.md @@ -1,251 +1,116 @@ -# `ecommerce` — a sample writeonce e-commerce app +# `ecommerce` — a sample writeonce **monorepo** -A storefront + checkout + live ops dashboard in **~300 lines of `.wo`**. Exercises the features that make `.wo` distinct from a plain REST app: **cross-paradigm ACID transactions**, **type-attached lifecycle triggers**, **link types with properties**, and a **live-updating operations table**. +Two apps (customer **storefront** + ops **admin**) sharing one database, built from a common pool of types + business logic. Mirrors the Nx / Angular workspace pattern: `apps/*` for deployable binaries, `shared/*` for libraries imported across apps. -> Like the [blog sample](../blog/), this is a **docs artifact** — illustrative `.wo` source showing the shape of a real `wo init`'d project. Toolchain specified in [`../../runtime/wo-language.md`](../../runtime/wo-language.md). +> This project is a **docs artifact** — illustrative `.wo` source showing what a production-shaped writeonce workspace looks like. The master plan for the compiler + client runtime + per-app build is at [`../../plan/ui/00-overview.md`](../../plan/ui/00-overview.md). Sub-phases UI/01–07 implement each piece. -## What's here - -| File | What it shows | -| --- | --- | -| [`types/product.wo`](./types/product.wo) | Relational scalars + embedded doc (`meta`, `inventory`) + computed field (`available`) + graph edge (`similar_to`) + inventory-low trigger | -| [`types/order.wo`](./types/order.wo) | Tagged union status, array-of-struct `line_items`, computed `total`, four lifecycle triggers setting timestamp columns atomically | -| [`types/customer.wo`](./types/customer.wo) | Role union + `multi Product via Purchase` (link with properties) + `backlink Order.customer` | -| [`types/purchase.wo`](./types/purchase.wo) | `link Customer -> Product` — a graph edge **type** with its own columns (`order`, `qty`, `unit_price`) | -| [`logic/checkout.wo`](./logic/checkout.wo) | The canonical cross-paradigm transaction: reserve inventory + insert order + create graph edge, atomic across all three engines | -| [`ui/admin_orders.wo`](./ui/admin_orders.wo) | **The live order-ops table** — role-gated, auto-subscribes, delta-in-place updates | -| [`ui/storefront.wo`](./ui/storefront.wo) | Customer-facing product list with live inventory | -| [`ui/order_tracker.wo`](./ui/order_tracker.wo) | Customer-facing order history, same live engine, policy-filtered source | -| [`app.wo`](./app.wo) | Route table, Admin/Ops bypass policy, idempotent `seed()` | -| [`tests/checkout_test.wo`](./tests/checkout_test.wo) | Three tests covering the atomic checkout, the abort-without-partial-state guarantee, and the live-subscription delta stream | - -## Project layout +## Layout ``` ecommerce/ -├── wo.toml -├── app.wo -├── types/ -│ ├── customer.wo -│ ├── product.wo -│ ├── order.wo -│ └── purchase.wo # link type — graph edge with properties -├── logic/ -│ └── checkout.wo # transactional functions (fn … in txn snapshot) -├── ui/ -│ ├── storefront.wo -│ ├── order_tracker.wo -│ └── admin_orders.wo # the live ops table -└── tests/ +├── wo.toml # workspace manifest (apps[] + shared[] + database) +├── README.md # this file +│ +├── shared/ # code imported by one or more apps +│ ├── types/ +│ │ ├── customer.wo # role union, policy, service rest expose +│ │ ├── product.wo # inventory + similar_to graph +│ │ ├── order.wo # tagged-union status + line_items array +│ │ └── purchase.wo # link Customer -> Product +│ ├── logic/ +│ │ ├── checkout.wo # fn checkout / mark_paid / mark_shipped +│ │ └── seed.wo # on-startup demo seed + admin-ops bypass policy +│ └── components/ # reusable .htmlx partials (forward-looking — UI track) +│ ├── layout.htmlx # page chrome shared across apps +│ ├── money.htmlx # {{> money amount=total}} +│ └── order-row.htmlx # used by both orders tables +│ +├── apps/ # one binary per app +│ ├── storefront/ # customer-facing +│ │ ├── wo.toml # listen :8080, connect WO_DB +│ │ ├── app.wo # routes: / → home, /product/:sku, /cart, /orders +│ │ └── ui/ +│ │ ├── home/ +│ │ │ └── home.wo # product list, live inventory +│ │ ├── product-detail/ # (future) +│ │ └── orders/ +│ │ └── orders.wo # customer's own orders +│ │ +│ └── admin/ # ops dashboard +│ ├── wo.toml # listen :8081, connect WO_DB +│ ├── app.wo # role: Admin | Ops; routes: /orders +│ └── ui/ +│ └── orders/ +│ └── orders.wo # live ops table with fulfillment actions +│ +└── tests/ # workspace-level integration └── checkout_test.wo ``` -## Run it +Each app's UI screens live in their own directory (`apps//ui//`) with the Angular-style one-directory-per-component pattern — `.wo` declarative spec today, `.htmlx` template + `.css` stylesheet once the UI track's sub-phase 01–02 land. + +## What the two apps share + +- **Types** (`shared/types/`). Both apps see the same `Customer` / `Product` / `Order` / `Purchase` definitions. Row-level policies inside each `type` block control who sees what — the storefront's authenticated customer sees their own orders; the admin app's Admin/Ops role sees everyone's. +- **Logic** (`shared/logic/`). The `checkout`, `mark_paid`, `mark_shipped`, and `release_inventory` functions in `shared/logic/checkout.wo` are callable from either app (subject to role policy). `shared/logic/seed.wo` runs once when the shared DB daemon starts. +- **Components** (`shared/components/`). `.htmlx` partials — layout chrome, money formatting, an order-row renderer — reusable from either app's templates. + +## What's **not** shared (per-app) + +- **`app.wo`** declares app-specific routes + role gate. Storefront has no `/admin/*` routes; admin has no `/cart` or `/product/:sku`. +- **`ui/`** is per-app. The storefront's `orders/orders.wo` (customer's own orders, filtered by session) and the admin's `orders/orders.wo` (all orders with fulfillment actions) are different screens — same underlying `Order` type, different UI + policy scope. +- Each app's `wo.toml` names its own listen port and its own DB API key. + +## Running it ```bash -$ cd docs/examples/ecommerce -$ wo run -[wo] parsing: 10 files, 4 types + 1 link type, 3 ui screens, 4 fns -[wo] compiling schema: 3 sql tables, 2 doc collections, 2 graph edge types -[wo] starting runtime (engine: in-memory, data_dir: ./data, isolation: snapshot) -[wo] on startup: seed() — 1 customer, 2 products -[wo] HTTP listening on :8080 +# 1. Start the shared DB daemon — headless, just the engine + wire protocol +wo db serve --data-dir ./data # listens on wo://127.0.0.1:5555 - GET /api/products list - GET /api/products/:id get - WS /api/products/live subscribe - GET /api/orders list - GET /api/orders/:id get - WS /api/orders/live subscribe - GET /api/customers/:id get - GET /api/customers/me me - PATCH /api/customers/:id update - WS /api/customers/live subscribe - POST /api/fn/checkout fn checkout(customer, product, qty) -> Order - POST /api/fn/mark_paid fn mark_paid(order) - POST /api/fn/mark_shipped fn mark_shipped(order) +# 2. Start each app, pointing at the daemon +WO_DB=wo://127.0.0.1:5555 \ + STOREFRONT_DB_KEY=$ADMIN_TOKEN \ + wo run apps/storefront # HTTP on :8080 - GET / ui.storefront - GET /product/:sku ui.product-detail - GET /orders ui.order-tracker - GET /admin/orders ui.admin-orders (Admin | Ops) +WO_DB=wo://127.0.0.1:5555 \ + ADMIN_DB_KEY=$ADMIN_TOKEN \ + wo run apps/admin # HTTP on :8081 ``` -> **Runtime model.** The engine is a single-threaded event loop today ([Phase 2 concurrency](../../runtime/database/02-wo-language.md#concurrency-model)). Snapshot isolation is trivially correct because there are no concurrent writers — the checkout, mark_paid, and mark_shipped fns run sequentially even when fired in quick succession. The throughput ceiling is ~one core (plenty for the sample); sharding across independent engine processes is the horizontal-scale path. +Browser: +- `http://localhost:8080/` → storefront home (product list, live inventory) +- `http://localhost:8081/orders` → admin live orders table -## Exercise the cross-paradigm checkout - -The `fn checkout(...)` in [`logic/checkout.wo`](./logic/checkout.wo) is the canonical Phase 2 test case: one transaction that mutates relational, document, and graph state atomically. +## Building binaries ```bash -# Place an order — one HTTP call runs the whole BEGIN ... COMMIT block -$ curl -X POST localhost:8080/api/fn/checkout \ - -H "Authorization: Bearer $CUSTOMER_TOKEN" \ - -d '{"customer":1, "product":2, "qty":3}' - -{"id":1, "status":"Pending", "total":5997, "line_items":[{...}], "placed_at":"..."} - -# Verify inventory was reserved (not yet decremented) -$ curl localhost:8080/api/products/2 -{"sku":"SKU-GIZMO", "inventory":{"on_hand":12, "reserved":3, "reorder_at":3}, "available":9, ...} - -# Verify the graph edge was created in the same transaction -$ curl localhost:8080/api/customers/1/purchased -[{"target":{"sku":"SKU-GIZMO"}, "order":1, "qty":3, "unit_price":1999, "at":"..."}] +wo build apps/storefront # → target/wo/storefront +wo build apps/admin # → target/wo/admin +wo build --all # everything in apps/ ``` -If the inventory check failed inside `checkout`, **none** of the above writes happen — the order isn't created, the reservation isn't made, and the graph edge doesn't exist. That atomicity is the whole point of building your own engine instead of stitching Postgres + Neo4j. +Each binary is a static ELF with only the app's own `app.wo` + `ui/` + the `shared/` it imports baked in. Drop any binary on a server next to a running `wo db serve` and it works. -## Watch the live admin ops table +## Stage 2 caveat -Open the admin orders UI in a browser: +The current runtime at [`crates/rt/`](../../../crates/rt/) is a single-process Stage 2 prototype — it doesn't yet know about: -```bash -$ open http://localhost:8080/admin/orders # authenticated as Admin or Ops -``` +- `[workspace]` manifests (per-app build — UI sub-phase 05) +- The `wo db serve` daemon split (UI sub-phase 06) +- Per-app `api_key_env` authorisation scope (UI sub-phase 07) +- `.htmlx` compilation from `##ui` blocks (UI sub-phases 01–03) +- Per-app policy composition (UI sub-phase 07) -The page renders a table with the columns declared in [`ui/admin_orders.wo`](./ui/admin_orders.wo). Behind the scenes, the client runtime has opened one WebSocket to the engine's subscription endpoint: +So `cargo run --bin wo -- run docs/examples/ecommerce` today walks the whole tree, finds every `.wo` file under `shared/` + `apps/`, parses the types, and serves the union REST API on :8080 — treating the monorepo as one giant app. Useful for exercising the types; not reflective of the production shape. See [`docs/plan/ui/00-overview.md`](../../plan/ui/00-overview.md) for the sub-phase sequence that gets each piece online. -``` -WS /api/orders/live ? status!=Cancelled -``` +## Comparison with the blog sample -Now, from another terminal, fire a sequence of state changes: +The [`blog` sample](../blog/) is still a single-app layout (`types/`, `ui/`, `logic/` at the root) because the blog has exactly one front-end surface — there's no customer-vs-admin split. Both patterns are first-class; pick based on whether your schema serves one app or many. A flat single-app layout is a degenerate workspace with one `apps/` entry. -```bash -# 1. New customer places an order — admin table gains a row, highlighted for 2s -$ curl -X POST localhost:8080/api/fn/checkout -d '{"customer":2,"product":1,"qty":1}' +## Source pointers -# 2. Payment webhook flips status Pending → Paid — row updates in place, paid_at fills in -$ curl -X POST localhost:8080/api/fn/mark_paid -d '{"order":2}' - -# 3. Ops ships the order — status → Shipped, shipped_at fills in -$ curl -X POST localhost:8080/api/fn/mark_shipped -d '{"order":2}' -``` - -The browser table re-renders each row delta as it arrives, without a full list refetch. `status` cell swaps its pill colour; timestamp cells populate. No polling anywhere in the path — the deltas are emitted by the transaction coordinator on commit, routed through the subscription registry, and pushed down the socket ([Phase 4](../../runtime/database/04-client-api.md)). - -A filtered subscription — the admin clicking the **"Ready to ship"** quick-filter — doesn't rebuild state client-side. It sends the new predicate to the server, which replies with a `SNAPSHOT` frame of just the matching rows, then streams deltas that match the new predicate. Also zero-polling. - -## Generate a typed Go client - -```bash -$ wo gen sdk --lang go --out ./client -[wo] reading types from ./types/ and fns from ./logic/ -[wo] writing ./client/sdk.go (4 types, 13 endpoints, 4 fns, 3 subscriptions) -``` - -The generated client speaks the native wire protocol: - -```go -import "myshop/client" - -c, _ := client.Connect(ctx, "wo://localhost:8080", client.WithToken(token)) - -// Typed transactional function call -order, err := c.Checkout(ctx, client.CheckoutArgs{ - Customer: 1, - Product: 2, - Qty: 3, -}) - -// Typed live subscription — same wire as the admin UI uses -sub, _ := c.Orders.Subscribe(ctx, client.Where{Status: client.Ne(client.Cancelled)}) -for d := range sub.C { - switch d.Kind { - case client.Insert: - fmt.Printf("new order #%d from %s — $%.2f\n", d.Row.ID, d.Row.Customer.Name, float64(d.Row.Total)/100) - case client.Update: - fmt.Printf("order #%d → %s\n", d.Row.ID, d.Row.Status) - } -} -``` - -## Checkout from Go without codegen - -For ad-hoc scripts, admin tools, or client paths not on the app's hot loop, send raw `.wo` DML with `client.Wo(...)`. The server parses the block exactly like `wo run` would — same parser, same transaction coordinator, same `RETURNING` alias table — so the **cross-paradigm checkout runs in one round trip**: - -```go -import "go.writeonce.dev/wo" - -c, _ := wo.Connect(ctx, "wo://localhost:8080", wo.WithToken(token)) - -// Same logic as fn checkout(), but authored at the Go call site. -// BEGIN SNAPSHOT ... COMMIT runs server-side; RETURNING aliases ($pid, $oid) -// thread from the SQL UPDATE/INSERT into the Cypher CREATE within the txn. -result, err := c.Wo(ctx, ` - BEGIN SNAPSHOT; - - UPDATE products - SET inventory.reserved = inventory.reserved + $qty - WHERE id = $pid AND available >= $qty - RETURNING id AS pid; - - INSERT INTO orders (customer, status, line_items) - VALUES ($uid, 'Pending', [{product: $pid, qty: $qty, unit_price: $unit}]) - RETURNING id AS oid; - - MATCH (u:Customer {id: $uid}), (p:Product {id: $pid}) - CREATE (u)-[:PURCHASED {order: $oid, qty: $qty, unit_price: $unit}]->(p); - - COMMIT; -`, wo.Params{"uid": 1, "pid": 2, "qty": 3, "unit": 4999}) - -if err != nil { log.Fatal(err) } -orderID := result.Aliases["oid"].(int64) -fmt.Printf("created order #%d\n", orderID) -``` - -**When to reach for this form** — see the [raw-vs-typed guidance in the Go SDK doc](../../runtime/database/05-go-sdk.md#when-to-use-raw-wo-vs-typed-codegen). Rule of thumb: typed `c.Checkout(...)` for the app's storefront; raw `c.Wo(...)` for an ops console that runs a custom report, or when you want to paste a block from [`logic/checkout.wo`](./logic/checkout.wo) straight into Go. - -## Run the tests - -```bash -$ wo test -=== tests/checkout_test.wo === - checkout atomically reserves inventory, creates order, and creates graph edge OK (8ms) - checkout aborts without partial state when inventory is insufficient OK (4ms) - admin live-orders subscription receives deltas across the order lifecycle OK (14ms) - -PASS 3/3 tests, 0 failures (26ms) -``` - -The third test is the important one for the docs: it proves that the same engine that serves `/admin/orders` in the browser delivers deltas in commit order through a programmatic `subscribe live` handle. One engine, one delta stream, two consumers (the browser and the test). - -## Build a production binary - -```bash -$ wo build --target linux-amd64 --out bin/shop -[wo] static binary: bin/shop (15 MB — database + HTTP + subscription engine embedded) -$ ./bin/shop -[wo] HTTP listening on :8080 -``` - -Drop the binary on a server, give it a writable directory for `./data/` (WAL + engine state), run it behind nginx or let it terminate TLS itself. The admin ops table works on the first page-load — no Redis, no Kafka, no separate DB process, no ORM-and-migration dance. - -## Compare to the blog sample - -Both projects use the same language and runtime. They showcase different slices: - -| Feature | [blog](../blog/) | ecommerce (this project) | -| --- | --- | --- | -| Embedded document | `article.meta` | `product.meta`, `product.inventory` | -| Graph edges (zero-prop) | tags, related | similar_to | -| Graph edges **with** properties | — | `type Purchase link Customer -> Product` | -| Tagged union | — | `Pending \| Paid \| Shipped \| ...` | -| Computed field | `word_count` | `available`, `total` (sum over line_items) | -| Array-of-struct column | — | `line_items: [{product, qty, unit_price}]` | -| Stored procedure (`fn ... in txn`) | seed only | full checkout + fulfillment | -| Cross-paradigm transaction | — | `checkout` (relational + doc + graph atomic) | -| Live subscription | list view | live **ops** table with in-place delta updates | -| Row-level policy | draft hiding | customer sees own orders; ops/admin sees all | - -If the blog shows **what a CRUD app looks like in `.wo`**, the ecommerce sample shows **what a transactional business app looks like in `.wo`** — and why building the engine as part of the language is the differentiator. - -## What to read next - -- [`../../runtime/wo-language.md`](../../runtime/wo-language.md) — language overview and toolchain -- [`../../runtime/database/02-wo-language.md`](../../runtime/database/02-wo-language.md) — the schema/query two-layer spec -- [`../../runtime/database/04-client-api.md`](../../runtime/database/04-client-api.md) — the wire protocol and subscription engine behind the live ops table -- [`../../runtime/database/06-lowcode-fullstack.md`](../../runtime/database/06-lowcode-fullstack.md) — `##ui` / `##app` block spec -- [`../../../prototypes/wo-db/tests/checkout.wo`](../../../prototypes/wo-db/tests/checkout.wo) — the C++ prototype's smoke test that exercises the same cross-paradigm transaction at the query layer +- **Master plan:** [`../../plan/ui/00-overview.md`](../../plan/ui/00-overview.md) +- **Language spec the `##ui`/`##app`/`policy` blocks obey:** [`../../runtime/database/06-lowcode-fullstack.md`](../../runtime/database/06-lowcode-fullstack.md) +- **Wire protocol the app binaries speak to the DB daemon:** [`../../runtime/database/04-client-api.md`](../../runtime/database/04-client-api.md) +- **v1 template engine that `.htmlx` compilation will reuse:** [`../../../reference/crates/wo-htmlx/`](../../../reference/crates/wo-htmlx/) +- **Checkout transaction that's the canonical cross-paradigm test:** [`shared/logic/checkout.wo`](shared/logic/checkout.wo) diff --git a/docs/examples/ecommerce/apps/admin/app.wo b/docs/examples/ecommerce/apps/admin/app.wo new file mode 100644 index 0000000..10a49fa --- /dev/null +++ b/docs/examples/ecommerce/apps/admin/app.wo @@ -0,0 +1,19 @@ +-- Admin app manifest: the ops/fulfillment dashboard binary. +-- Entire app is gated to role `Admin | Ops`; the storefront's public routes +-- don't live here at all. The cross-entity "admin/ops bypass row filters" +-- policy is global (shared/logic/seed.wo) — this app inherits it via the +-- shared DB connection. + +##app +name: "admin" +version: 1 +theme: "dark" +i18n: [en] +role: Admin | Ops -- app-level gate + +routes: + /orders -> ui.orders -- apps/admin/ui/orders/ + -- future: + -- /inventory -> ui.inventory + -- /customers -> ui.customers + -- /reports -> ui.reports diff --git a/docs/examples/ecommerce/ui/admin_orders.wo b/docs/examples/ecommerce/apps/admin/ui/orders/orders.wo similarity index 77% rename from docs/examples/ecommerce/ui/admin_orders.wo rename to docs/examples/ecommerce/apps/admin/ui/orders/orders.wo index b9bde75..b64217f 100644 --- a/docs/examples/ecommerce/ui/admin_orders.wo +++ b/docs/examples/ecommerce/apps/admin/ui/orders/orders.wo @@ -3,13 +3,19 @@ -- shown; every commit (checkout, status flip, trigger-updated timestamp) pushes -- a delta down the WebSocket and the client runtime swaps the row in place. -- No polling, no refresh button. +-- +-- Lives in the admin app; the app's own `app.wo` gates it to role Admin | Ops, +-- so customers never hit this route. Same underlying Order table as the +-- storefront's customer-facing order-tracker — row-level policies decide who +-- sees which rows; delta fanout is cross-app (see docs/plan/ui/00-overview.md +-- § Sub-phase 13). ##ui -#admin-orders +#orders title: "Orders — Live" source: Order live: true - role: Admin | Ops -- gated by role; customers can't reach /admin/orders + role: Admin | Ops -- secondary gate (app-level gate in app.wo) -- Sidebar filter controls bind to these predicates at render time. filter: @@ -37,10 +43,10 @@ sort: default: placed_at desc - -- Single-row + bulk actions. Each one calls a `fn` from logic/checkout.wo - -- (or a stdlib action like `email`). The compiler wires them to endpoints. + -- Single-row + bulk actions. Each one calls a `fn` from + -- shared/logic/checkout.wo or an app-local fulfillment fn. actions: - row-click: /admin/orders/:id + row-click: /orders/:id row-mark-paid: mark_paid(self) role: Admin | Ops row-mark-shipped: mark_shipped(self) role: Admin | Ops row-cancel: update self set status = Cancelled role: Admin diff --git a/docs/examples/ecommerce/apps/admin/wo.toml b/docs/examples/ecommerce/apps/admin/wo.toml new file mode 100644 index 0000000..7ba7029 --- /dev/null +++ b/docs/examples/ecommerce/apps/admin/wo.toml @@ -0,0 +1,24 @@ +name = "admin" +version = "0.1.0" +description = "Ops/fulfillment dashboard binary. Gated to role Admin | Ops." +app_kind = "app" + +[runtime] +wo = ">= 0.1" + +[dependencies] +shared = [ + "../../shared/types", + "../../shared/logic", + "../../shared/components", +] + +[server] +listen = ":8081" # admin runs on a separate port from storefront + +[database] +# Same daemon as storefront, different API key → daemon's policy table gives +# this app Admin-level row visibility (per docs/plan/ui/07-per-app-policies.md +# when that phase lands). +url = "wo://127.0.0.1:5555" +api_key_env = "ADMIN_DB_KEY" diff --git a/docs/examples/ecommerce/apps/storefront/app.wo b/docs/examples/ecommerce/apps/storefront/app.wo new file mode 100644 index 0000000..0cd3cfa --- /dev/null +++ b/docs/examples/ecommerce/apps/storefront/app.wo @@ -0,0 +1,19 @@ +-- Storefront app manifest: the customer-facing ecommerce binary. +-- Routes serve to anonymous visitors + authenticated customers; +-- admin/ops users exist as DB rows but have no route in this binary. +-- +-- The app binary connects to the shared `wo db` daemon (see +-- ../../wo.toml [database] + apps/storefront/wo.toml WO_DB env var). +-- Row-level policies on shared/types/*.wo scope what each session sees. + +##app +name: "storefront" +version: 1 +theme: "light" +i18n: [en] + +routes: + / -> ui.home -- apps/storefront/ui/home/ + /product/:sku -> ui.product-detail { key: $sku } -- apps/storefront/ui/product-detail/ + /cart -> ui.cart + /orders -> ui.orders -- apps/storefront/ui/orders/ diff --git a/docs/examples/ecommerce/ui/storefront.wo b/docs/examples/ecommerce/apps/storefront/ui/home/home.wo similarity index 50% rename from docs/examples/ecommerce/ui/storefront.wo rename to docs/examples/ecommerce/apps/storefront/ui/home/home.wo index 3caae4a..c574be5 100644 --- a/docs/examples/ecommerce/ui/storefront.wo +++ b/docs/examples/ecommerce/apps/storefront/ui/home/home.wo @@ -1,9 +1,15 @@ --- Public product list. Live inventory — when a checkout reserves the last --- unit, the "In stock" badge flips to "Out of stock" on every open browser --- without refresh. +-- Storefront home — public product list. Live inventory: when a checkout +-- reserves the last unit, the "In stock" badge flips to "Out of stock" on +-- every open browser without refresh. +-- +-- Angular-component-style layout: home/{home.wo, home.htmlx, home.css}. +-- home.wo is the declarative spec; home.htmlx (when added) lets an author +-- hand-tune the template while the compiler still generates the +-- + wo:bind subscription glue. See +-- [docs/plan/ui/00-overview.md](../../../../../plan/ui/00-overview.md). ##ui -#storefront +#home title: "Shop" source: Product live: true diff --git a/docs/examples/ecommerce/ui/order_tracker.wo b/docs/examples/ecommerce/apps/storefront/ui/orders/orders.wo similarity index 98% rename from docs/examples/ecommerce/ui/order_tracker.wo rename to docs/examples/ecommerce/apps/storefront/ui/orders/orders.wo index 7d37007..2372468 100644 --- a/docs/examples/ecommerce/ui/order_tracker.wo +++ b/docs/examples/ecommerce/apps/storefront/ui/orders/orders.wo @@ -3,7 +3,7 @@ -- to `customer == $session.user`, so a user only ever sees their own orders. ##ui -#order-tracker +#orders title: "Your Orders" source: Order{ customer == $session.user } live: true diff --git a/docs/examples/ecommerce/apps/storefront/wo.toml b/docs/examples/ecommerce/apps/storefront/wo.toml new file mode 100644 index 0000000..21f1634 --- /dev/null +++ b/docs/examples/ecommerce/apps/storefront/wo.toml @@ -0,0 +1,26 @@ +name = "storefront" +version = "0.1.0" +description = "Customer-facing ecommerce binary. Connects to the shared wo-db daemon." +app_kind = "app" # not a library; produces a binary under target/wo/storefront + +[runtime] +wo = ">= 0.1" + +# The storefront depends on the shared type + logic sources one workspace +# level up. Paths are workspace-relative. +[dependencies] +shared = [ + "../../shared/types", + "../../shared/logic", + "../../shared/components", +] + +[server] +listen = ":8080" # what this app exposes to browsers + +[database] +# Pointer to the shared DB daemon. At runtime, `WO_DB=wo://...` overrides. +url = "wo://127.0.0.1:5555" +# The app presents this API key on connect; the daemon's per-app policy +# table scopes what rows it can see. +api_key_env = "STOREFRONT_DB_KEY" diff --git a/docs/examples/ecommerce/shared/components/layout.htmlx b/docs/examples/ecommerce/shared/components/layout.htmlx new file mode 100644 index 0000000..f42bb01 --- /dev/null +++ b/docs/examples/ecommerce/shared/components/layout.htmlx @@ -0,0 +1,42 @@ +{{!-- + Top-level page chrome shared across every app in the workspace. + Both storefront and admin inherit this layout and override the `{{> slot.content}}` + partial with their own screen's body. + + The `wo:bind="session.user.name"` in the header is a field-level live binding: + when the customer updates their profile, every open tab's header repaints + without a page reload — the same delta-dispatch mechanism the live tables use, + just on a singleton subscription. + + See docs/plan/ui/01-htmlx-format-spec.md (to land in a future sub-phase) + for the full grammar. +--}} + + + + + {{page.title}} — {{app.name}} + + {{> slot.head}} + + +
+ {{app.name}} + + {{#if session.user}} + {{session.user.name}} + {{/if}} +
+ +
+ {{> slot.content}} +
+ +
+ Built with writeonce · {{app.version}} +
+ + {{!-- Client runtime + subscription manifest are injected automatically. --}} + + + diff --git a/docs/examples/ecommerce/shared/components/money.htmlx b/docs/examples/ecommerce/shared/components/money.htmlx new file mode 100644 index 0000000..7102c9c --- /dev/null +++ b/docs/examples/ecommerce/shared/components/money.htmlx @@ -0,0 +1,6 @@ +{{!-- + Tiny partial: renders a Money value (minor units — cents) as a localised + currency string. Called from any screen via `{{> money amount=total}}`. + See apps/admin/ui/orders/orders.wo columns block for a live example. +--}} +${{divide amount by 100}}.{{mod amount by 100 padleft 2 with "0"}} diff --git a/docs/examples/ecommerce/shared/components/order-row.htmlx b/docs/examples/ecommerce/shared/components/order-row.htmlx new file mode 100644 index 0000000..e507632 --- /dev/null +++ b/docs/examples/ecommerce/shared/components/order-row.htmlx @@ -0,0 +1,23 @@ +{{!-- + One order row — used by both apps' order tables. Different columns are + displayed depending on the caller, via conditional sub-partials; the + `wo:bind="status"` attribute makes the status cell live-update on delta + frames regardless of which table embeds this row. + + Customer view (storefront/ui/orders/orders.wo): {{> order-row for="customer"}} + Admin view (admin/ui/orders/orders.wo): {{> order-row for="ops"}} +--}} + + {{id}} + {{status}} + {{#if (eq for "ops")}} + {{customer.name}} + {{customer.email}} + {{/if}} + {{> money amount=total}} + {{relative placed_at}} + {{#if (eq for "ops")}} + {{relative paid_at}} + {{relative shipped_at}} + {{/if}} + diff --git a/docs/examples/ecommerce/logic/checkout.wo b/docs/examples/ecommerce/shared/logic/checkout.wo similarity index 100% rename from docs/examples/ecommerce/logic/checkout.wo rename to docs/examples/ecommerce/shared/logic/checkout.wo diff --git a/docs/examples/ecommerce/app.wo b/docs/examples/ecommerce/shared/logic/seed.wo similarity index 69% rename from docs/examples/ecommerce/app.wo rename to docs/examples/ecommerce/shared/logic/seed.wo index e24b840..4456da2 100644 --- a/docs/examples/ecommerce/app.wo +++ b/docs/examples/ecommerce/shared/logic/seed.wo @@ -1,24 +1,17 @@ -##app -name: "ecommerce" -version: 1 -theme: "light" -i18n: [en] +-- Idempotent demo seed + the cross-entity Admin/Ops bypass policy. +-- Both belong to the *shared* layer rather than to any one app: +-- * The seed primes the shared database so either the storefront or the +-- admin app shows something on first boot. +-- * The bypass policy applies across every type, so it has to live where +-- every app can see it. +-- Runs once when the shared `wo db serve` daemon comes up — each app connects +-- over the wire and inherits the already-seeded state. -routes: - / -> ui.storefront - /product/:sku -> ui.product-detail { key: $sku } - /cart -> ui.cart - /orders -> ui.order-tracker - /admin/orders -> ui.admin-orders role: Admin | Ops - --- Cross-entity policy — Admin + Ops bypass row-level filters so they can see --- every customer's orders in the live ops table. policy admin-ops-bypass applies_to: Order, Product, Customer, Purchase when: $session.role == Admin or $session.role == Ops effect: skip-row-filters --- Startup seed: idempotent demo data so `wo run` shows something useful. on startup do: seed() diff --git a/docs/examples/ecommerce/types/customer.wo b/docs/examples/ecommerce/shared/types/customer.wo similarity index 100% rename from docs/examples/ecommerce/types/customer.wo rename to docs/examples/ecommerce/shared/types/customer.wo diff --git a/docs/examples/ecommerce/types/order.wo b/docs/examples/ecommerce/shared/types/order.wo similarity index 100% rename from docs/examples/ecommerce/types/order.wo rename to docs/examples/ecommerce/shared/types/order.wo diff --git a/docs/examples/ecommerce/types/product.wo b/docs/examples/ecommerce/shared/types/product.wo similarity index 100% rename from docs/examples/ecommerce/types/product.wo rename to docs/examples/ecommerce/shared/types/product.wo diff --git a/docs/examples/ecommerce/types/purchase.wo b/docs/examples/ecommerce/shared/types/purchase.wo similarity index 100% rename from docs/examples/ecommerce/types/purchase.wo rename to docs/examples/ecommerce/shared/types/purchase.wo diff --git a/docs/examples/ecommerce/wo.toml b/docs/examples/ecommerce/wo.toml index aaf6aa0..b8b0866 100644 --- a/docs/examples/ecommerce/wo.toml +++ b/docs/examples/ecommerce/wo.toml @@ -1,16 +1,32 @@ -name = "ecommerce" +name = "ecommerce-workspace" version = "0.1.0" -description = "A sample writeonce e-commerce app: cross-paradigm ACID checkout + live order-ops table" +description = "Monorepo: shared types + logic powering two apps (storefront, admin) against one DB" +kind = "workspace" # not a single app — see [workspace] below [runtime] wo = ">= 0.1" -[server] -listen = ":8080" +# Apps = one binary each; shared = libraries imported across apps by path. +# Mirrors Angular/Nx workspaces (apps/* + libs/*). See +# docs/plan/ui/00-overview.md § Target layout. +[workspace] +apps = [ + "apps/storefront", + "apps/admin", +] +shared = [ + "shared/types", + "shared/logic", + "shared/components", +] +# The shared DB daemon — one process, no UI, just the engine + wire protocol. +# Starts with `wo db serve` (see docs/plan/ui/06-shared-db-daemon.md when +# that sub-phase lands). Apps connect at runtime via WO_DB. [database] +listen = "127.0.0.1:5555" # native wire-protocol port data_dir = "./data" -isolation = "snapshot" # snapshot isolation for the canonical checkout flow +isolation = "snapshot" # default for cross-paradigm checkout [test] -parallel = false # ordering tests touch the same rows; serialize +parallel = false # cross-app integration tests hit the same DB diff --git a/docs/plan/09-concurrency-scaleout.md b/docs/plan/09-concurrency-scaleout.md new file mode 100644 index 0000000..6336300 --- /dev/null +++ b/docs/plan/09-concurrency-scaleout.md @@ -0,0 +1,125 @@ +# 09 — Scale-out: thread-per-core for 10k concurrent users + +**Context sources:** [`./08-sendfile-static-assets.md`](./08-sendfile-static-assets.md) (last single-threaded phase), [`./assembly/02-writeonce-stance.md`](./assembly/02-writeonce-stance.md) (the "single-threaded" policy we're now refining), [`../runtime/database/02-wo-language.md#concurrency-model`](../runtime/database/02-wo-language.md#concurrency-model) (original concurrency stance), [`docs/examples/ecommerce/`](../examples/ecommerce/) (the target workload), [`./linux/`](./linux/) (kernel primitives), [`reference/go/src/runtime/`](../../reference/go/src/runtime/) (precedent for a runtime that scales across threads). + +## Context + +Phases 02–08 produce a single-threaded event-loop runtime with zero external Rust dependencies — good enough for the blog sample and the first 500k ops/sec on one core. **The ecommerce workload at `docs/examples/ecommerce/` pushes past that ceiling**: ~10,000 connected websocket subscribers watching `/api/orders/live`, ~1,000 checkouts per second during peak, and every commit fanning out delta frames to a sizeable subset of the connected clients. No single core survives that, regardless of how tight the event loop is. + +This phase is the refinement of the Phase-2 concurrency doctrine "shard to scale past one core" into a concrete architecture. The stance stays the same — **no Go-style goroutines, no work-stealing across threads, no shared mutable heap** — but now we have multiple event loops, each owning its core, its share of connections, and its slice of engine state. + +This doc is a **master plan**. It outlines sub-phases 10–15 at a high level; each sub-phase lands as its own numbered plan doc when implementation starts. No code changes in this pass. + +## Goal + +Serve the ecommerce sample at 10,000 concurrent websocket subscribers + 1,000 checkouts/second on a single 8–16 core box, with per-request tail latencies (`p99`) inside 50 ms for reads and 100 ms for commits. After this phase sequence lands, `cargo run --bin wo -- run docs/examples/ecommerce` can handle production-shaped load with only the process count as the horizontal scale knob. + +## Design decisions (locked) + +1. **Thread-per-core, not M:N.** `N` OS threads pinned to `N` cores via `sched_setaffinity(cpu_set_t)`. Each thread runs its own event loop (the [phase-02 `runtime/` module](./02-event-loop-epoll.md)) plus a local shard of engine state. Pinned for the thread's lifetime; a connection accepted on thread K stays on thread K forever. Precedent: Seastar / ScyllaDB / Redis Cluster. +2. **Shared-nothing state.** No cross-thread mutable access to the catalog, engine rows, or subscription registry. Communication is message-passing over single-producer-single-consumer ring buffers (crossbeam-style, built on `std::sync::atomic`, per [`./assembly/02-writeonce-stance.md`](./assembly/02-writeonce-stance.md) — still no asm). If thread A needs to touch data owned by thread B, it sends a message; B processes it on its own tick. +3. **SO_REUSEPORT for listener-side load balancing.** Every thread binds a socket with `SO_REUSEPORT` on the same `:8080` — the kernel distributes incoming SYNs across the `N` listener sockets with consistent hashing on the connection 4-tuple. No user-space accept-thread bottleneck. Linux ≥ 3.9 is fine; ≥ 4.5 adds `BPF` filters for custom routing if we ever need session-affinity. +4. **Per-thread io_uring ring.** Each thread gets its own `io_uring_setup` ring with `IORING_SETUP_SINGLE_ISSUER` + `IORING_SETUP_SQPOLL` ([per `./linux/07-io_uring.md`](./linux/07-io_uring.md)). No ring sharing across threads — simpler ordering, no contention. +5. **Shard key: customer id (modulo N).** The ecommerce schema is customer-centric — one customer's orders + purchase edges + cart live on the same shard. Cross-customer queries (admin `list orders`) fan out; same-customer operations (checkout) are local. Blog shard key would be `author.id` for the same reason. +6. **Cross-shard transactions via 2PC.** A checkout that updates inventory on shard A and customer balance on shard B uses two-phase commit between the two engine threads. Phase 4's transaction coordinator (from [`../runtime/database/02-wo-language.md`](../runtime/database/02-wo-language.md) § Cross-Paradigm Transaction Coordinator) already handles this pattern for sql+doc+graph inside one process; it generalises cleanly to cross-thread. +7. **No Go-style goroutines.** Connections are not tasks that migrate. Each connection's state machine runs on its owning thread's event loop, just as it does in the single-threaded model — the difference is there are now `N` event loops running concurrently. + +## What we copy from Go, what we don't + +Read [`reference/go/src/runtime/netpoll_epoll.go`](../../reference/go/src/runtime/netpoll_epoll.go) and [`reference/go/src/runtime/proc.go`](../../reference/go/src/runtime/proc.go) for the shape; copy the **ideas** about fd-to-loop mapping and atomic-counter-based wake-up. Do **not** copy: + +| Go feature | Why writeonce skips it | +| --- | --- | +| Goroutines (M:N scheduling, work stealing) | Goroutines pay context-switch + GC-scan costs the thread-per-core model avoids. Scylla benchmarks consistently beat Go-style runtimes at the same hardware. | +| Shared heap + GC | No heap GC — Rust ownership. Data is partitioned across threads, not shared with locks. | +| `gogo` / `mcall` / `systemstack` asm | No scheduler-controlled stack switching. See [`./assembly/02-writeonce-stance.md`](./assembly/02-writeonce-stance.md). | +| `cgo` boundary | Rust is the only language. `libc` is already ABI-compatible via `extern "C"`. | +| `asyncPreempt` preemption | Handlers run to completion on their owning thread. Back-pressure comes from bounded per-thread queues, not preemption. | + +And what we **do** copy: + +| Go pattern | Writeonce translation | +| --- | --- | +| Per-P netpoller (the `pp.pollDesc` model) | Per-thread `EventLoop` (the phase-02 `runtime::EventLoop`) | +| `netpollBreak` (fd wake-up via sendto) | Per-thread `eventfd` — one fd per thread, write to it to wake a sleeping `epoll_wait`. See [`./linux/02-eventfd.md`](./linux/02-eventfd.md). | +| `findrunnable` (what to do when idle) | Per-thread idle-state: drain in-process message queues, run compaction, run periodic timers (from [`./linux/03-timerfd.md`](./linux/03-timerfd.md)). | +| `runtime.GOMAXPROCS` | `WO_THREADS` env var (defaults to `std::thread::available_parallelism()`). | + +## Linux primitives this phase leans on (beyond the phase-02/03/08 set) + +Reference cards already exist for most; this phase adds the ones that are cross-thread-specific: + +| Primitive | Use | Reference | +| --- | --- | --- | +| `SO_REUSEPORT` | N listener sockets on the same port; kernel load-balances accepts | [`reference/linux/net/core/sock_reuseport.c`](../../reference/linux/net/core/sock_reuseport.c) — worth adding `linux/12-so-reuseport.md` | +| `sched_setaffinity` + `cpu_set_t` | Pin each thread to its core | [`reference/linux/kernel/sched/core.c`](../../reference/linux/kernel/sched/core.c) | +| `futex(2)` | Fallback cross-thread wait if per-thread eventfd wake-up isn't enough | [`reference/linux/kernel/futex/`](../../reference/linux/kernel/futex/) — worth `linux/13-futex.md` | +| `membarrier(2)` | Process-wide memory barrier when a rebalance migrates state between threads | [`reference/linux/kernel/sched/membarrier.c`](../../reference/linux/kernel/sched/membarrier.c) | +| `io_uring` with `IORING_SETUP_SINGLE_ISSUER` | One ring per thread, pinned | [`./linux/07-io_uring.md`](./linux/07-io_uring.md) | +| `eventfd` per thread | Cross-thread wake-up — thread A writes to thread B's eventfd to deliver a message | [`./linux/02-eventfd.md`](./linux/02-eventfd.md) | +| `mmap(MAP_HUGETLB)` | Per-thread arena allocator backed by 2 MB pages for cache locality | [`./linux/08-mmap.md`](./linux/08-mmap.md) | + +## Sub-phase sequence + +Each one lands as its own numbered plan doc when ready for implementation. Smoke test (`cargo run --bin wo -- run docs/examples/ecommerce` serves correctly) stays green after every sub-phase. + +### `10-thread-per-core.md` — N event loops, `SO_REUSEPORT` + +Introduce a thread-pool manager at `crates/rt/src/runtime/scheduler.rs` (Go parallel: `proc.go`). Spawn `WO_THREADS` OS threads at boot; each pins itself and runs an `EventLoop`. Replace the single `Listener` with per-thread listeners bound `SO_REUSEPORT` to the same port. State is still global at first (shared `Arc>`) — one thing at a time. Exit criterion: `wo run` boots N threads visible in `ps -T`, accepts load balanced across them per `ss -tnp`, no regression in the 20-assertion blog smoke. + +### `11-sharded-engine.md` — per-thread engine state + +Partition the in-memory engine catalog + row BTreeMaps by shard id (= thread id). Shard key is `customer.id` for ecommerce / `author.id` for blog / per-type default for anything else. Add a shard router in front of every REST/WS handler: resolve the shard from the request's identifying field, send an in-process message to that thread's mailbox, await response. Shared `Arc>` goes away; each thread owns its slice. Cross-shard reads (admin `list orders`) fan out to every thread and merge results. + +### `12-per-shard-wal.md` — one WAL file per shard + +Each thread has its own `foo.wal` + `foo.data` + per-thread `io_uring` ring (phase 3's durability work, repeated per shard). Recovery is parallel across threads. No shared WAL writer thread. Group commit is per-thread. + +### `13-cross-shard-subscriptions.md` — LIVE fanout + +A commit on shard K that creates/updates rows of type T needs to wake subscribers on every shard watching T. Via broadcast: K writes the delta to a per-subscriber-thread mailbox — one message per destination thread, not per subscriber. The destination thread then does the fine-grained predicate match against its local subscription table. Avoids N² traffic when N connections watch the same stream. + +### `14-cross-shard-txn.md` — 2PC for transactions that span shards + +`fn checkout(customer, product, qty)` might touch shards A (customer), B (product), and C (order) if they hash differently. The transaction coordinator (already designed in [`../runtime/database/02-wo-language.md`](../runtime/database/02-wo-language.md) § Cross-Paradigm Transaction Coordinator) generalises to cross-shard: `begin(snapshot_ts)` broadcasts to all participating shards, `prepare()` collects votes, `commit(wal_lsn)` atomically flips markers, `abort()` if any participant refuses. The per-shard WAL entries carry the 2PC state machine. + +### `15-observability-and-rebalance.md` — ops + +Per-shard metrics (connections, ops/s, p99, WAL lag), Prometheus scrape endpoint on one well-known thread. A `WO_RESHARD` admin command migrates a contiguous customer-id range from shard K to shard K′ via state snapshot → replay → cutover. For a fixed-core deployment this is rare; matters when `WO_THREADS` changes between runs. + +## Verification targets (after 15 lands) + +Ecommerce sample on an 8-core box with `WO_THREADS=8`: + +| Metric | Target | How measured | +| --- | --- | --- | +| Concurrent WS subscribers | **10,000** | `websocat` fan-out against `/api/orders/live` + persistent count | +| Checkout throughput | **1,000/s** sustained | Load driver fires `POST /api/fn/checkout` with per-customer key distribution | +| Read p99 | **< 50 ms** | `GET /api/orders?customer=X` under 10k-subscriber background load | +| Commit p99 | **< 100 ms** | Measured from `POST /api/fn/checkout` acceptance to HTTP ack | +| Memory steady-state | **< 2 GB RSS** | 10k connections × 2 KB/conn + engine working set | +| Dep count | **1** (`libc`) | `crates/rt/Cargo.toml` still has only libc after all this | +| `wo run docs/examples/blog` | still boots and serves | phase 02–08 regression test, unchanged | + +## Non-scope + +- **No Go-style goroutines, even after this phase.** Adding M:N scheduling is not on the roadmap. When one core runs out, add more cores (more threads) — horizontally, thread-per-core. +- **No distributed (multi-node) sharding.** This phase is single-box only. Redis-Cluster-style network sharding is a separate future phase; the in-process shard bus (phase 10's mailboxes) is not the same thing as a cluster membership protocol. +- **No dynamic thread count at runtime.** `WO_THREADS` is set at boot and pinned. Adding/removing a thread means a rolling restart. Acceptable for a database; fundamental to the zero-contention model. +- **No work-stealing.** A slow handler on thread A does not get rebalanced to thread B. Back-pressure is the thread-local queue filling up. If one thread hot-spots because of a bad shard key, the fix is to reshard — not to steal. +- **No `std::thread::available_parallelism` on exotic hosts.** `WO_THREADS` override covers kubernetes CFS-bound pods, NUMA partitioning, and single-core debug runs. +- **No new external Rust dependencies.** Same stance as phases 02–08 — `libc` only. Message passing, atomics, affinity, futex — all through libc or `std::sync::atomic`. + +## Escape hatch + +If the "single core per process, shard across processes" argument ([Redis Cluster model](../runtime/database/02-wo-language.md#concurrency-model)) turns out to be more operationally attractive than a single multi-threaded process, **every decision in this plan translates**. Per-thread shards become per-process shards; `SO_REUSEPORT` inside the kernel becomes a reverse proxy in front; in-process mailboxes become Unix domain sockets. The phase-02 event loop is the reusable atom regardless. + +## Cross-references + +- [`./08-sendfile-static-assets.md`](./08-sendfile-static-assets.md) — last prerequisite phase; feature-complete single-threaded runtime. +- [`./assembly/02-writeonce-stance.md`](./assembly/02-writeonce-stance.md) — updated to reference this phase's thread-per-core model; still no asm. +- [`../runtime/database/02-wo-language.md#concurrency-model`](../runtime/database/02-wo-language.md#concurrency-model) — the stance this plan refines. +- [`reference/go/src/runtime/proc.go`](../../reference/go/src/runtime/proc.go) — Go's scheduler, for contrast. +- [`reference/go/src/runtime/netpoll_epoll.go`](../../reference/go/src/runtime/netpoll_epoll.go) — per-P netpoller, the idea we borrow. +- [`reference/linux/net/core/sock_reuseport.c`](../../reference/linux/net/core/sock_reuseport.c) — kernel load balancer. +- [`reference/linux/kernel/sched/core.c`](../../reference/linux/kernel/sched/core.c) — affinity syscalls. diff --git a/docs/plan/assembly/02-writeonce-stance.md b/docs/plan/assembly/02-writeonce-stance.md index 86b9383..da3dad8 100644 --- a/docs/plan/assembly/02-writeonce-stance.md +++ b/docs/plan/assembly/02-writeonce-stance.md @@ -8,8 +8,8 @@ Each Go asm category from [`01-go-runtime-asm.md`](./01-go-runtime-asm.md) maps | Go asm need | What writeonce uses | Why it covers the gap | | --- | --- | --- | -| Scheduler stack switching (`gogo`, `mcall`, `systemstack`) | — nothing — | Single-threaded event loop (see Phase 2 [Concurrency Model](../runtime/database/02-wo-language.md#concurrency-model)). No goroutines, no stack switching, no `g0`. | -| Preemption (`asyncPreempt`) | — nothing — | No preemption. Handlers run to completion on the single thread. | +| Scheduler stack switching (`gogo`, `mcall`, `systemstack`) | — nothing — | Single-threaded event loop through phases 02–08 (see Phase 2 [Concurrency Model](../runtime/database/02-wo-language.md#concurrency-model)). [Phase 09](../09-concurrency-scaleout.md) introduces **thread-per-core** scaling for the 10k-user ecommerce workload — but still no Go-style stack switching: each thread runs its own event loop, connections are pinned for their lifetime, and cross-thread work is message-passing, not scheduler-stealing. No goroutines, no `g0`, even at scale. | +| Preemption (`asyncPreempt`) | — nothing — | No preemption through phases 02–08. Phase 09's thread-per-core model keeps this property: handlers run to completion on whichever thread owns their connection. | | Atomic operations (`Load`, `Store`, `Cas`, `Xadd`, ...) | [`std::sync::atomic`](https://doc.rust-lang.org/std/sync/atomic/) | The compiler emits the right instruction per target — `LOCK CMPXCHG` on x86, `LDXR/STXR` on ARM, `LR.W/SC.W` on RISC-V. Ordering is in the type signature (`Ordering::Acquire`, `Release`, `SeqCst`). | | Memory barriers (`MFENCE` etc.) | [`std::sync::atomic::fence(Ordering)`](https://doc.rust-lang.org/std/sync/atomic/fn.fence.html) | One call, one fence, arch-neutral. | | `memmove` / `memequal` / `memclr` | [`core::ptr::copy`](https://doc.rust-lang.org/core/ptr/fn.copy.html), `<[T]>::copy_from_slice`, `==`, `[T]::fill(0)` | LLVM emits the same vectorised code Go's asm does, often better because it knows alignment statically. | @@ -45,6 +45,7 @@ Phase 3's WAL loop batches commits and fsyncs them in one `io_uring` submission. - **Rust answer:** [`std::hint::spin_loop()`](https://doc.rust-lang.org/std/hint/fn.spin_loop.html). The compiler emits `PAUSE` / `WFE` per target. Use `core::hint::black_box` to prevent compiler over-optimisation of measurement. - **When to escalate:** when benchmarks show a specific hotspot the compiler is provably wrong about. Not before. + ## The escape hatch If all three defences above are exhausted — benchmarked, documented, reviewed — assembly goes in: diff --git a/docs/plan/ui/00-overview.md b/docs/plan/ui/00-overview.md new file mode 100644 index 0000000..9af5520 --- /dev/null +++ b/docs/plan/ui/00-overview.md @@ -0,0 +1,213 @@ +# UI track — `.htmlx` live templates + Angular-style monorepo + +**Context sources:** [`docs/examples/ecommerce/ui/`](../../examples/ecommerce/ui/) (current `##ui` screens — storefront, order_tracker, admin_orders), [`docs/examples/ecommerce/types/`](../../examples/ecommerce/types/) + [`docs/examples/ecommerce/logic/`](../../examples/ecommerce/logic/) (the shared-schema + shared-fn anchor), [`reference/crates/wo-htmlx/`](../../../reference/crates/wo-htmlx/) (v1 template engine — `{{path}}`, `{{#each}}`, `{{> partial}}`, `data-bind` attributes), [`templates/`](../../../templates/) (v1 blog's concrete `.htmlx` usage), [`docs/runtime/database/06-lowcode-fullstack.md`](../../runtime/database/06-lowcode-fullstack.md) (Phase 6's `##ui` + `##app` block spec). + +## Context + +Three threads converge into one plan: + +1. **`##ui` needs a concrete output format.** Phase 6's spec says screens "compile to a render tree" served as SSR HTML with a thin client runtime, but the actual template format isn't named. The v1 `.htmlx` engine at [`reference/crates/wo-htmlx/`](../../../reference/crates/wo-htmlx/) already speaks `{{bindings}}`, `{{#each}}`, `{{> partials}}`, and `data-bind` attributes — it's 90% of what the new runtime needs and already has a working parser + renderer. Adopting it (and extending it with live-subscription semantics) is cheaper than inventing a new format. + +2. **The samples want a home that matches how real frontends are organised.** The ecommerce sample today is one flat directory with `types/`, `logic/`, and `ui/` beside each other. A real deployment has *multiple apps* against the same data: a customer storefront, an admin dashboard, a fulfillment console, maybe a read-only analytics viewer. Each has its own routes, its own policies, its own ideal binary shape. Angular (via Nx / Angular CLI workspaces) solved this with `apps/*` + `libs/*` on top of a shared root config — writeonce adopts the same shape. + +3. **Each app wants to be its own binary but share a database.** Running storefront and admin as one monolith conflates concerns: a CPU spike in admin stalls customer checkout; an admin auth bug opens customer data paths. Splitting into per-app binaries that share a single database backend (via the Phase-4 native wire protocol) gives blast-radius isolation without duplicating data. + +Intended outcome: `docs/examples/ecommerce/` refactors into a workspace with `shared/` + `apps/storefront/` + `apps/admin/`. `wo build apps/storefront` produces a `storefront` binary. `wo db serve` runs the shared database. The apps connect over `wo://…` and serve `.htmlx` SSR pages that subscribe to LIVE queries without a page reload. + +## Goal + +After this track's sub-phases land: + +- A **monorepo workspace** at `docs/examples/ecommerce/` with `shared/{types,logic,components}` + `apps/{storefront,admin}` structure. +- A **per-app binary** for each app under `apps/`: `wo build apps/storefront` → `./target/wo/storefront`, `wo build apps/admin` → `./target/wo/admin`. Each binary includes only its own `##ui` / `##app` / app-local types and logic; shared code compiles in by path reference. +- A **shared DB daemon** (`wo db serve`) — one process, no UI, just the engine and wire-protocol server. Each app binary connects as a client via the Phase-4 native protocol. +- **`.htmlx` as the compiled UI output.** Every `##ui` screen compiles into an `.htmlx` template file that the app binary serves; hand-written `.htmlx` files under `apps/X/ui/*.htmlx` are accepted as a first-class authoring alternative. +- **`.htmlx` subscribes.** A `` subtree in the template registers a LIVE query at page load; a ~20 KB client JS runtime patches DOM nodes on delta frames without reloading the page. +- **Per-app users + policies.** Each app declares its role set in `apps/X/app.wo`; row-level policies on shared types stay global, app-scoped policies layer on top per route. + +## Design decisions (locked) + +1. **`.htmlx` is the template format; `##ui` is the DSL that emits it.** Authors choose per-screen: declare `##ui #home { source: Product, columns: [...] }` in `.wo` and let the compiler produce `home.htmlx`; OR hand-write `home.htmlx` for a custom page. Both flow through the same `wo-htmlx` renderer. +2. **Extend v1 `.htmlx` with two new constructs** — `...` (subscription subtree) and `wo:bind="field"` (field-level live binding). The rest of the v1 syntax (`{{path}}`, `{{#each}}`, `{{> partial}}`) carries through unchanged. +3. **One binary per app, shared database process.** Not a shared library, not a monolith. Apps connect via the Phase-4 wire protocol (`wo://host:port`) — the same connection a Go/TS client would use. No in-process shared state between apps; their isolation is enforced by the OS process boundary. +4. **Angular-parallel workspace layout.** `apps/*` for deployable binaries, `shared/*` for libs shared across apps (types, logic, UI components), `wo.toml` at the workspace root. Each app also has its own `wo.toml` that names which `shared/` directories it depends on. +5. **File structure mirrors Angular component organisation.** Each UI screen lives in its own directory: `apps/storefront/ui/home/{home.wo, home.htmlx, home.css}`. Tests go in `home.test.wo`. Matches the Angular component pattern (`home.component.ts`, `home.component.html`, `home.component.scss`). +6. **Per-app routes, not per-screen routes.** `apps/storefront/app.wo` declares route table; each route maps to a `ui.` declared under `apps/storefront/ui/*/`. Cross-app navigation is an external redirect, not an internal route. +7. **Policy composition.** Global policies live in `shared/types/.wo` next to the `type` declaration (today). App-scoped policies live in `apps/X/app.wo` and AND with the global set — an admin app might relax a storefront policy for ops roles but can never relax beyond what the type's own policy permits. + +## Angular parallels — what writeonce copies, what it doesn't + +| Angular feature | Writeonce translation | Notes | +| --- | --- | --- | +| `nx workspace` / `angular.json` | Root `wo.toml` with `[workspace] apps = [...], shared = [...]` | Path references, not package registry | +| `apps//` | `apps//` with `app.wo` + `ui/` + local `types/` + local `logic/` | 1:1 naming | +| `libs//` | `shared//` | Used `shared/` instead of `libs/` — matches the more common monorepo idiom (Nx defaults to `libs`, but `shared` is clearer for this audience) | +| `.ts` + `.html` + `.scss` | `.wo` + `.htmlx` + `.css` under `ui//` | One-directory-per-screen | +| `ng build ` | `wo build apps/` | Per-app binary output | +| Dependency injection | Service-block resolution — `service rest` blocks in shared types are callable from any app by import | No runtime DI container; bindings are resolved at compile time | +| RxJS observables | LIVE subscription frames on a WebSocket | Declarative `live` attribute instead of imperative `.subscribe(...)` | +| Zone.js change detection | Per-row delta dispatch + field-level `wo:bind` | No full-tree change detection — only the rows/fields the delta names get repainted | +| `HttpClient` | Built-in wire-protocol client inside the app binary | No separate library to import; always present | + +### What we don't copy + +- **No TypeScript.** Authoring is `.wo` (for logic) + `.htmlx` (for templates) + `.css`. If a page needs bespoke JS interactivity beyond what `wo:bind` covers, a `