From 7a2e09cd198511ce3d77706ee7a85d1a1c04ecc5 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Sun, 12 Jul 2026 04:25:49 +0200 Subject: [PATCH] Update plan with kanban --- docs/plan/00-kanban.md | 79 ++++++++++++++++++++++ docs/plan/05-hand-rolled-json.md | 2 + docs/plan/06-bespoke-error.md | 2 + docs/plan/07-inotify-content-watcher.md | 2 + docs/plan/08-sendfile-static-assets.md | 2 + docs/plan/09-concurrency-scaleout.md | 2 + docs/plan/10-storage-foundations.md | 2 + docs/plan/11-wal-and-recovery.md | 2 + docs/plan/12-engine-disk-cutover.md | 2 + docs/plan/13-class-model-live-pricing.md | 2 + docs/plan/14-mvc-ui-implementation.md | 2 + docs/plan/exploration/c-runtime/00-plan.md | 2 + 12 files changed, 101 insertions(+) create mode 100644 docs/plan/00-kanban.md diff --git a/docs/plan/00-kanban.md b/docs/plan/00-kanban.md new file mode 100644 index 0000000..1db71e9 --- /dev/null +++ b/docs/plan/00-kanban.md @@ -0,0 +1,79 @@ +# Plan kanban โ€” backend: runtime, database, API + +Status board for every phase doc under `docs/plan/`. **Focus: the backend** โ€” the runtime, the database engine, and the REST/`.wo`-language API. Frontend phases are parked, not deleted. Each phase file carries a matching status banner; this board is the index. + +Statuses: โœ… **done** ยท ๐Ÿ”„ **in progress** ยท โฌœ **not started** ยท โธ **parked (out of backend focus)** + +## Board + +### Track 1 โ€” Runtime foundations (dependency removal, kernel primitives) + +| Status | Phase | Doc | Notes | +| --- | --- | --- | --- | +| โœ… | 01 crate scaffolding | [done/01](done/01-scafolding-crates.md) | 15 crates | +| โœ… | 02 epoll event loop | [done/02](done/02-event-loop-epoll.md) | `runtime/netpoll_epoll.rs` | +| โœ… | 03 hand-rolled HTTP | [done/03](done/03-hand-rolled-http.md) | + keep-alive & pipelining (follow-up under plan 09) | +| โœ… | 04 tokio/axum cutover | [done/04](done/04-cutover-remove-tokio-axum.md) | deps now: anyhow, serde, serde_json, libc | +| โฌœ | 05 hand-rolled JSON | [05](05-hand-rolled-json.md) | removes serde/serde_json โ€” **next in this track** | +| โฌœ | 06 bespoke error type | [06](06-bespoke-error.md) | removes anyhow | +| โฌœ | 07 inotify content watcher | [07](07-inotify-content-watcher.md) | `wo dev` hot reload | +| โฌœ | 08 sendfile static assets | [08](08-sendfile-static-assets.md) | needed by parked UI track too | + +### Track 2 โ€” Concurrency & scale-out (plan 09) โ€” ๐Ÿ”„ in progress + +| Status | Sub-phase | Notes | +| --- | --- | --- | +| โœ… | 09a thread-per-core | `runtime/scheduler.rs`, `SO_REUSEPORT`, pinned `wo-shard-` workers | +| โœ… | 09b sharded engine | `shard.rs` bus; `Arc>` deleted; interleaved ids; fan-out lists | +| โœ… | 09c per-shard WAL | `wal.rs`; ack-after-fsync; boot replay; `meta` shard guard | +| โœ… | โ€” keep-alive (follow-up) | reads ร—3.4 โ†’ 770k/s; C phase-C sequence | +| โœ… | โ€” io_uring group commit (follow-up) | `netpoll_io_uring.rs` raw ring; 4.7ร— durable writes on real disk | +| โฌœ | 09d cross-shard subscriptions | LIVE fan-out, one message per shard โ€” pairs with Stage 3 | +| โฌœ | 09e cross-shard transactions (2PC) | needed by `fn checkout` spanning shards | +| โฌœ | 09f observability & reshard | per-shard metrics, `WO_RESHARD` | + +All numbers + find-and-fix stories: [09-concurrency-scaleout.md](09-concurrency-scaleout.md) shipped notes and the [benchmark table](../../prototypes/wo-rt-c/README.md). + +### Track 3 โ€” Storage & durability (plans 10โ€“12) + +| Status | Phase | Doc | Notes | +| --- | --- | --- | --- | +| โฌœ | 10 storage foundations | [10](10-storage-foundations.md) | scope reduced: WAL framing/fallocate landed via 09c | +| โฌœ | 11 WAL & recovery | [11](11-wal-and-recovery.md) | remaining: snapshots (`.data`), compaction, WAL rotation โ€” replay core shipped in 09c | +| โฌœ | 12 engine disk cutover | [12](12-engine-disk-cutover.md) | mmap arena engine (C phase B is the proving ground) | + +### Track 4 โ€” Language & API (`.wo` on the wire) โ€” ๐Ÿ”„ in progress + +| Status | Phase | Doc | Notes | +| --- | --- | --- | --- | +| โœ… | 13a class surface | [13](13-class-model-live-pricing.md) | `class` parses, CRUD serves, spec amended | +| โฌœ | 13b method execution | [13](13-class-model-live-pricing.md) | `POST /api//:id/`, row-scoped txn โ€” **the next API milestone** | +| โฌœ | 13c LIVE pricing push | [13](13-class-model-live-pricing.md) | Stage 3 scoped: subscription registry, WS at `/api//live`, replaces the 501 stub | +| โฌœ | Stage 3 wire layer | [../runtime/database/04-client-api.md](../runtime/database/04-client-api.md) | full subscription engine + wire protocol; 13c is its beachhead | +| โฌœ | 13e pricing at scale | [13](13-class-model-live-pricing.md) | wires demo to 09; hot-row reads | + +Ecommerce sample status (verified 2026-06-13, `api.rest` **17/17 expected statuses pass**): route wiring, empty lists, 405 for un-exposed ops, 404 for unregistered routes, 501 Stage-3 stubs โ€” all exactly as documented. `fn checkout`, `on startup` seeding, and status-lifecycle triggers await 13b-style execution + 09e. + +### Track 5 โ€” C proving ground โ€” โœ… done (Aโ€“F) + +| Status | Phase | Doc | +| --- | --- | --- | +| โœ… | A threads ยท B arena ยท C io_uring ยท D WAL ยท E recovery ยท F bench+ACID | [exploration/c-runtime/00-plan.md](exploration/c-runtime/00-plan.md) | + +859k reads/s, 618k durable commits/s; found the ack-ordering + fd-ABA bugs the Rust port then avoided. Optional phase G (splice `wo-db` C++ engine) remains an idea. + +### Track 6 โ€” Frontend โ€” โธ parked (out of backend focus) + +| Status | Phase | Doc | Notes | +| --- | --- | --- | --- | +| โธ | 13d pricing UI | [13](13-class-model-live-pricing.md) | MVC triplet exists as design artifact | +| โธ | 14 MVC UI implementation (14aโ€“f) | [14](14-mvc-ui-implementation.md) | htmlx engine, SCSS subset, controllers, SSR, actions, live patching | +| โธ | UI exploration track | [exploration/ui/00-overview.md](exploration/ui/00-overview.md) | 00โ€“08 design docs stay current | + +## Suggested order of play (backend) + +1. **13b โ€” method execution** (unblocks `fn checkout` semantics, the ecommerce sample's core promise) +2. **13c / Stage 3 LIVE** with **09d** fan-out (turns every 501 stub real; the ecommerce order-ops board's backend) +3. **09e โ€” 2PC** (cross-shard `fn checkout` โ€” the canonical ACID demo end-to-end) +4. **05/06** dependency removal (mechanical, any time) +5. **10โ€“12** storage completion (snapshots/compaction; arena engine) diff --git a/docs/plan/05-hand-rolled-json.md b/docs/plan/05-hand-rolled-json.md index ee3317e..f3e4ca7 100644 --- a/docs/plan/05-hand-rolled-json.md +++ b/docs/plan/05-hand-rolled-json.md @@ -1,5 +1,7 @@ # 05 โ€” Hand-Rolled JSON +> **Kanban: โฌœ not started** โ€” Track 1 (runtime foundations), next in the dependency-removal sequence. Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./04-cutover-remove-tokio-axum.md`](./04-cutover-remove-tokio-axum.md), [`../../prototypes/wo-db/src/value.hpp`](../../prototypes/wo-db/src/value.hpp). ## Goal diff --git a/docs/plan/06-bespoke-error.md b/docs/plan/06-bespoke-error.md index 3184d4c..3ff8d96 100644 --- a/docs/plan/06-bespoke-error.md +++ b/docs/plan/06-bespoke-error.md @@ -1,5 +1,7 @@ # 06 โ€” Bespoke Error Type +> **Kanban: โฌœ not started** โ€” Track 1 (runtime foundations). Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./05-hand-rolled-json.md`](./05-hand-rolled-json.md), [`../01-problem.md`](../01-problem.md). ## Goal diff --git a/docs/plan/07-inotify-content-watcher.md b/docs/plan/07-inotify-content-watcher.md index 21c1b6d..481b653 100644 --- a/docs/plan/07-inotify-content-watcher.md +++ b/docs/plan/07-inotify-content-watcher.md @@ -1,5 +1,7 @@ # 07 โ€” `inotify` Content Watcher +> **Kanban: โฌœ not started** โ€” Track 1 (runtime foundations). Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./02-event-loop-epoll.md`](./02-event-loop-epoll.md), [`./linux/00-linux.md`](./linux/00-linux.md) ยง File Watching, [`../02-recovery.md`](../02-recovery.md) ยง No AWS Infrastructure. ## Goal diff --git a/docs/plan/08-sendfile-static-assets.md b/docs/plan/08-sendfile-static-assets.md index 71ee4e3..5aa7921 100644 --- a/docs/plan/08-sendfile-static-assets.md +++ b/docs/plan/08-sendfile-static-assets.md @@ -1,5 +1,7 @@ # 08 โ€” `sendfile` Zero-Copy Static Serving +> **Kanban: โฌœ not started** โ€” Track 1 (runtime foundations); also a prerequisite of the parked UI track. Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./03-hand-rolled-http.md`](./03-hand-rolled-http.md), [`./linux/00-linux.md`](./linux/00-linux.md) ยง Efficient File Serving, [`../02-recovery.md`](../02-recovery.md). ## Goal diff --git a/docs/plan/09-concurrency-scaleout.md b/docs/plan/09-concurrency-scaleout.md index c39b9aa..eed28bd 100644 --- a/docs/plan/09-concurrency-scaleout.md +++ b/docs/plan/09-concurrency-scaleout.md @@ -1,5 +1,7 @@ # 09 โ€” Scale-out: thread-per-core for 10k concurrent users +> **Kanban: ๐Ÿ”„ in progress** โ€” 09a/09b/09c โœ… shipped (+ keep-alive and io_uring group-commit follow-ups, measured in the shipped notes below); 09d/09e/09f โฌœ not started. Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./08-sendfile-static-assets.md`](./08-sendfile-static-assets.md) (last single-threaded phase), [`./assembly/02-writeonce-stance.md`](./assembly/02-writeonce-stance.md) (the "single-threaded" policy we're now refining), [`../runtime/database/02-wo-language.md#concurrency-model`](../runtime/database/02-wo-language.md#concurrency-model) (original concurrency stance), [`docs/examples/ecommerce/`](../examples/ecommerce/) (the target workload), [`./linux/`](./linux/) (kernel primitives), [`reference/go/src/runtime/`](../../reference/go/src/runtime/) (precedent for a runtime that scales across threads). ## Context diff --git a/docs/plan/10-storage-foundations.md b/docs/plan/10-storage-foundations.md index c836a80..4a5ffd2 100644 --- a/docs/plan/10-storage-foundations.md +++ b/docs/plan/10-storage-foundations.md @@ -1,5 +1,7 @@ # 10 โ€” Storage Foundations: on-disk row codec + segment append path +> **Kanban: โฌœ not started (scope reduced)** โ€” WAL framing/fallocate/CRC landed early via plan 09c. Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./done/04-cutover-remove-tokio-axum.md`](./done/04-cutover-remove-tokio-axum.md), [`../runtime/database/03-inmemory-engine.md`](../runtime/database/03-inmemory-engine.md), [`../runtime/database/07-wo-seg-migration.md`](../runtime/database/07-wo-seg-migration.md), [`./exploration/postgresql/smgr-and-md.md`](./exploration/postgresql/smgr-and-md.md), [`./exploration/postgresql/page-format.md`](./exploration/postgresql/page-format.md), [`./exploration/linux/12-pwrite-fsync.md`](./exploration/linux/12-pwrite-fsync.md), [`./exploration/linux/09-fallocate.md`](./exploration/linux/09-fallocate.md), [`reference/crates/wo-seg/src/`](../../reference/crates/wo-seg/src/). ## Goal diff --git a/docs/plan/11-wal-and-recovery.md b/docs/plan/11-wal-and-recovery.md index b0e7ffe..264af7d 100644 --- a/docs/plan/11-wal-and-recovery.md +++ b/docs/plan/11-wal-and-recovery.md @@ -1,5 +1,7 @@ # 11 โ€” WAL + crash recovery +> **Kanban: โฌœ not started (scope reduced)** โ€” replay + ack-after-fsync + group commit landed via 09c and its follow-ups; remaining here: snapshots (`.data`), compaction, WAL rotation. Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./10-storage-foundations.md`](./10-storage-foundations.md), [`../runtime/database/02-wo-language.md#concurrency-model`](../runtime/database/02-wo-language.md#concurrency-model), [`../runtime/database/03-inmemory-engine.md`](../runtime/database/03-inmemory-engine.md), [`./exploration/postgresql/wal.md`](./exploration/postgresql/wal.md), [`./exploration/postgresql/buffer-and-checkpoint.md`](./exploration/postgresql/buffer-and-checkpoint.md), [`./exploration/linux/12-pwrite-fsync.md`](./exploration/linux/12-pwrite-fsync.md), [`../02-recovery.md`](../02-recovery.md). ## Goal diff --git a/docs/plan/12-engine-disk-cutover.md b/docs/plan/12-engine-disk-cutover.md index e7b1c05..db801ea 100644 --- a/docs/plan/12-engine-disk-cutover.md +++ b/docs/plan/12-engine-disk-cutover.md @@ -1,5 +1,7 @@ # 12 โ€” Engine cutover: rows live on disk +> **Kanban: โฌœ not started** โ€” the C prototype's phase B (mmap arena) is the proving ground. Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./10-storage-foundations.md`](./10-storage-foundations.md), [`./11-wal-and-recovery.md`](./11-wal-and-recovery.md), [`../runtime/database/03-inmemory-engine.md`](../runtime/database/03-inmemory-engine.md), [`../runtime/database/07-wo-seg-migration.md`](../runtime/database/07-wo-seg-migration.md), [`./exploration/postgresql/buffer-and-checkpoint.md`](./exploration/postgresql/buffer-and-checkpoint.md), [`./exploration/postgresql/page-format.md`](./exploration/postgresql/page-format.md), [`./exploration/linux/12-pwrite-fsync.md`](./exploration/linux/12-pwrite-fsync.md). ## Goal diff --git a/docs/plan/13-class-model-live-pricing.md b/docs/plan/13-class-model-live-pricing.md index 994ff3b..20c8d0c 100644 --- a/docs/plan/13-class-model-live-pricing.md +++ b/docs/plan/13-class-model-live-pricing.md @@ -1,5 +1,7 @@ # 13 โ€” Class model + live pricing: state and methods, no inheritance +> **Kanban: ๐Ÿ”„ in progress** โ€” 13a โœ… shipped; 13b (method execution) is the next API milestone; 13c โฌœ; 13d โธ parked (frontend); 13e โฌœ. Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`../runtime/database/02-wo-language.md`](../runtime/database/02-wo-language.md) (schema layer, ยง Schema-Layer DML brace disambiguation, ยง Cross-Paradigm Transaction Coordinator), [`../runtime/database/04-client-api.md`](../runtime/database/04-client-api.md) (subscription engine), [`./09-concurrency-scaleout.md`](./09-concurrency-scaleout.md) (thread-per-core scale-out), [`./exploration/ui/00-overview.md`](./exploration/ui/00-overview.md) + [`./exploration/ui/01-htmlx-format-spec.md`](./exploration/ui/01-htmlx-format-spec.md) (live UI), [`../examples/pricing/`](../examples/pricing/) (the demo this phase makes real), [`../examples/ecommerce/shared/logic/checkout.wo`](../examples/ecommerce/shared/logic/checkout.wo) (the existing `fn โ€ฆ in txn snapshot` signature style methods reuse). ## Context diff --git a/docs/plan/14-mvc-ui-implementation.md b/docs/plan/14-mvc-ui-implementation.md index 12aef8b..4ef7817 100644 --- a/docs/plan/14-mvc-ui-implementation.md +++ b/docs/plan/14-mvc-ui-implementation.md @@ -1,5 +1,7 @@ # 14 โ€” MVC UI implementation: model = class, view = htmlx + scss, controller = .wo +> **Kanban: โธ parked (frontend)** โ€” backend focus first; design stays current. Board: [00-kanban.md](00-kanban.md) + **Context sources:** [`./exploration/ui/08-mvc-structure.md`](./exploration/ui/08-mvc-structure.md) (the design this plan implements), [`./exploration/ui/01-htmlx-format-spec.md`](./exploration/ui/01-htmlx-format-spec.md) / [`02-ui-compiler.md`](./exploration/ui/02-ui-compiler.md) / [`03-client-runtime.md`](./exploration/ui/03-client-runtime.md) (the three UI-track pieces this plan sequences, each with port sources and LOC budgets), [`./13-class-model-live-pricing.md`](./13-class-model-live-pricing.md) (the class methods controllers call: 13a/13b; the LIVE deltas views consume: 13c), [`../examples/pricing/ui/pricing/`](../examples/pricing/ui/pricing/) (the reference MVC triplet), [`reference/crates/wo-htmlx/`](../../reference/crates/wo-htmlx/) (the v1 template engine, primary port source). ## Context diff --git a/docs/plan/exploration/c-runtime/00-plan.md b/docs/plan/exploration/c-runtime/00-plan.md index 2fce655..e51655f 100644 --- a/docs/plan/exploration/c-runtime/00-plan.md +++ b/docs/plan/exploration/c-runtime/00-plan.md @@ -1,5 +1,7 @@ # wo-rt-c roadmap โ€” multi-threaded io_uring RAM database runtime, in C +> **Kanban: โœ… done** โ€” phases Aโ€“F all shipped with measured exit evidence below. Board: [../../00-kanban.md](../../00-kanban.md) + **Context sources:** [`prototypes/wo-rt-c/wo-rt.c`](../../../../prototypes/wo-rt-c/wo-rt.c) (phase 0 โ€” the single-threaded epoll baseline), [`../../09-concurrency-scaleout.md`](../../09-concurrency-scaleout.md) (the thread-per-core doctrine every phase here miniaturizes), [`../../10-storage-foundations.md`](../../10-storage-foundations.md) / [`11-wal-and-recovery.md`](../../11-wal-and-recovery.md) / [`12-engine-disk-cutover.md`](../../12-engine-disk-cutover.md) (the storage track), kernel reference cards [`../linux/07-io_uring.md`](../linux/07-io_uring.md), [`08-mmap.md`](../linux/08-mmap.md), [`09-fallocate.md`](../linux/09-fallocate.md), [`12-pwrite-fsync.md`](../linux/12-pwrite-fsync.md), [`02-eventfd.md`](../linux/02-eventfd.md). ## Goal