update docs

This commit is contained in:
shoney.arickathil 2026-08-18 02:04:58 +02:00
parent 7274a6f758
commit eb61c97dfb
3 changed files with 90 additions and 138 deletions

View file

@ -91,9 +91,10 @@ Story slice: [`docs/stories/language-runtime-database/07-logwatcher-proof.md`](s
- Iterations 8–12 (shard-actor runtime, database engine, `@table`/query, HTTP
layer, fibers, blue-green): unchanged, and unblocked by this plan.
Two tracks run in this repo. The critical path is the **language track**:
iterations 3 → 4 → 5 → 6 → 7, ending at _compile and run log-watcher_. The
Rust-runtime track is shipped-and-maintained, not advancing.
The project is the **language track**: iterations 3 → 4 → 5 → 6 → 7, ending at
_compile and run log-watcher_, then the database engine (9/9b) and beyond. (The
prior Rust `wo` runtime was removed from the repo 2026-08-18 — see
[`discarded.md`](plan/discarded.md).)
---
@ -293,27 +294,9 @@ recorded, not silently owed:
for story iteration 7b). Spec success criterion 3 is now **MET**;
`just oop-accept` passes all five criteria.
### Rust runtime track — Stage 2 shipped, maintained
| Status | Phase | Doc | Notes |
| ------ | ------------------------ | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| ✅ | 01 crate scaffolding | [done/01](plan/done/01-scafolding-crates.md) | 15 crates |
| ✅ | 02 epoll event loop | [done/02](plan/done/02-event-loop-epoll.md) | `runtime/netpoll_epoll.rs` |
| ✅ | 03 hand-rolled HTTP | [done/03](plan/done/03-hand-rolled-http.md) | + keep-alive & pipelining |
| ✅ | 04 tokio/axum cutover | [done/04](plan/done/04-cutover-remove-tokio-axum.md) | deps now: anyhow, serde, serde_json, libc |
| ✅ | 09a thread-per-core | [09](plan/09-concurrency-scaleout.md) | `scheduler.rs`, `SO_REUSEPORT`, pinned `wo-shard-<t>` workers |
| ✅ | 09b sharded engine | [09](plan/09-concurrency-scaleout.md) | `shard.rs` bus; `Arc<Mutex<Engine>>` deleted; interleaved ids |
| ✅ | 09c per-shard WAL | [09](plan/09-concurrency-scaleout.md) | ack-after-fsync; boot replay; `meta` shard guard |
| ✅ | — keep-alive follow-up | [09](plan/09-concurrency-scaleout.md) | reads ×3.4 → 770k/s |
| ✅ | — io_uring group commit | [09](plan/09-concurrency-scaleout.md) | raw ring; 4.7× durable writes on real disk |
| ✅ | 16a PG wire client | [16](plan/16-postgres-mirror.md) | hand-rolled protocol v3, zero crates |
| ✅ | 16b PG backup mirror | [16](plan/16-postgres-mirror.md) | async JSONB upserts behind the WAL ack; RAM authoritative |
| ✅ | 13a class surface | [13](plan/13-class-model-live-pricing.md) | `class` parses, CRUD serves |
| ✅ | 13b method execution | [13](plan/13-class-model-live-pricing.md) | row-scoped txn per call; abort → 409 rollback |
| ✅ | — `@table` + indexed DML | [13](plan/13-class-model-live-pricing.md) | secondary indexes, `find_by`, `select Type{…}`, REST filters |
| ✅ | C proving ground A–F | [exploration/c-runtime/00-plan.md](plan/exploration/c-runtime/00-plan.md) | 859k reads/s, 618k durable commits/s; found the ack-ordering + fd-ABA bugs the Rust port avoided |
Ecommerce sample (verified 2026-06-13): `api.rest` 17/17 expected statuses pass.
The C proving-ground work (`exploration/c-runtime/`, phases A–F: 859k reads/s,
618k durable commits/s) fed the current C runtime and remains as an
[exploration study](plan/exploration/c-runtime/00-plan.md).
---
@ -358,22 +341,6 @@ log-watcher proof.
- `is` — cut 2026-08-10, 0 uses in the driving workload; emptied plan 8's old
Task 7, which is deleted rather than deferred
### Rust runtime track — not advancing while the language track runs
| Status | Phase | Doc | Notes |
| ------ | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| ⬜ | 05 hand-rolled JSON | [05](plan/05-hand-rolled-json.md) | removes serde/serde_json |
| ⬜ | 06 bespoke error type | [06](plan/06-bespoke-error.md) | removes anyhow |
| ⬜ | 07 inotify content watcher | [07](plan/07-inotify-content-watcher.md) | `wo dev` hot reload |
| ⬜ | 08 sendfile static assets | [08](plan/08-sendfile-static-assets.md) | needed by the parked UI track |
| ⬜ | 09d cross-shard subscriptions | [09](plan/09-concurrency-scaleout.md) | LIVE fan-out; pairs with Stage 3 |
| ⬜ | 09e cross-shard transactions (2PC) | [09](plan/09-concurrency-scaleout.md) | needed by `fn checkout` spanning shards |
| ⬜ | 09f observability & reshard | [09](plan/09-concurrency-scaleout.md) | per-shard metrics, `WO_RESHARD` |
| ⬜ | 10–12 storage completion | [10](plan/10-storage-foundations.md), [11](plan/11-wal-and-recovery.md), [12](plan/12-engine-disk-cutover.md) | snapshots, compaction, WAL rotation, mmap arena engine |
| ⬜ | 13c LIVE pricing push · Stage 3 wire layer · 13e at scale | [13](plan/13-class-model-live-pricing.md) | replaces the 501 stub |
| ⬜ | 15a–15e MCP over streamable HTTP | [15](plan/15-mcp-streamable-http.md) | 15e needs 13c + 09d |
| ⬜ | 16c–16f typed columns, lossless resync, restore, SCRAM | [16](plan/16-postgres-mirror.md) | |
### Frontend — removed as stale (2026-08-17)
The `##ui` / `.htmlx` LiveView frontend track — 13d pricing UI, the 14-MVC-UI

View file

@ -1,138 +1,122 @@
# 08 — Project structure: the writeonce monorepo
Canonical map of the repository: what every root directory is, who writes to it, and where it is headed as the OOP + systems tracks land. Companion to [`CLAUDE.md`](../CLAUDE.md) (working rules) and the two track specs ([`superpowers/specs/2026-08-01-oop-compiler-vm-design.md`](superpowers/specs/2026-08-01-oop-compiler-vm-design.md), [`superpowers/specs/2026-08-01-systems-track-design.md`](superpowers/specs/2026-08-01-systems-track-design.md)). Plan documents govern the *target* entries; this doc is the one place the whole shape is visible.
Canonical map of the repository: what every root directory is and who writes to
it. Companion to [`CLAUDE.md`](../CLAUDE.md) (working rules), the status board
([`00-status.md`](00-status.md)), and the story arc
([`stories/language-runtime-database/00-story.md`](stories/language-runtime-database/00-story.md)).
writeonce is **one compiled language, one runtime, one embedded database, one
binary**: OCaml `woc` compiles `.wo` source to a `.wob` image, the C `wovm`
runs it, and `woc <dir>` bakes the two into a single standalone executable. The
prior Rust `wo` runtime was removed 2026-08-18 (see
[`plan/discarded.md`](plan/discarded.md)); the tree below is only the current
project.
## The one-page map
```
writeonce-all/
├── compiler/ ⏳ OCaml `woc` — lexer→parser→types→owner→emit (plans 2, 8)
├── runtime/ ✅ C runtime — wo-rt.c event-loop reference + ⏳ src/ wovm VM (plans 1, 4–7, 9)
├── crates/ ✅ Rust runtime `rt` + 14 phase-scaffold crates — active until C-stack parity
├── tests/ ⏳ corpus/ — conformance fixtures: run / compile-fail / trap / gc / sys / actor / db
├── scripts/ ⏳ oop-e2e.sh, oop-parity.sh — corpus + parity harnesses (plan 3)
├── prototypes/ ✅ wo-db C++ query-engine reference; ⚠ wo-rt-c = stale duplicate of runtime/
├── .dev/reference/ ✅ v1 crates workspace, colibri, llama-cpp; linux/ + go/ symlinks (per-dev)
├── docs/ ✅ ALL documentation: numbered design docs, plan/, runtime/, examples/, superpowers/
├── justfile recipes: rt-c-demo/bench today; woc-/wovm-/oop-accept as plans land
├── Cargo.toml the Rust workspace root (crates/*; .dev/reference/crates excluded)
└── .dev/ gitignored per-developer links + commit.md draft (see .dev/README.md)
├── compiler/ OCaml `woc` — lexer→parser→types→owner→emit; produces the compiler binary
├── runtime/ C `wovm` — the register VM that runs .wob images (src/); phase A–F C reference (wo-rt.c, bench/)
├── database/ C embedded engine — class-shaped tables, secondary indexes, typed WAL + recovery
├── tests/ corpus/ — conformance fixtures: run / compile-fail / trap / gc
├── scripts/ oop-e2e.sh (corpus runner), mkdist.sh / install-accept.sh (packaging), sample acceptance
├── docs/ ALL documentation: numbered docs, stories/, plan/, examples/, superpowers/
├── .dev/ gitignored per-developer links + reference study trees (v1 crates, colibri, llama-cpp)
├── justfile task runner: woc-/wovm-build, the *-test gates, oop-accept, dist, install-accept
├── VERSION single-sourced toolchain version (stamped into woc/wovm; asserted by `just dist`)
└── README.md the getting-started front door (also the writeonce.de landing content)
```
✅ exists today · ⏳ created by a named plan · ⚠ cleanup note (below).
## Root directories in detail
### `compiler/` — the OCaml `woc` compiler (target)
Created by plan 2 (`docs/plan/compiler/2026-08-01-woc-compiler-front.md`), grown by plans 3 and 8.
### `compiler/` — the OCaml `woc` compiler
```
compiler/
├── dune-project
├── README.md orientation: pipeline map, build/test commands
├── plan/ compiler-track docs: architecture.md + plans 2, 3, 8
│ (recorded exception to the docs-under-docs/ rule)
├── plan/ compiler-track docs: architecture.md + the woc plans
├── src/ one module per stage: diag, token, lexer, ast, parser,
│ types, owner, emit, disasm, dump
├── bin/main.ml the woc executable (check/emit/build modes)
├── bin/main.ml the woc executable (check / --emit / build / version modes)
└── test/ golden runner + golden/ fixtures per stage
```
Doctrine: OCaml stdlib only — no Menhir, no ppx, no opam libraries; handwritten lexer and recursive-descent parser. Architecture map with the marked reference-study paths: [`docs/plan/compiler/architecture.md`](plan/compiler/architecture.md).
Doctrine: OCaml stdlib only — no Menhir, no ppx, no opam libraries; handwritten
lexer and recursive-descent parser. Build: `just woc-build`; gate:
`just woc-test`. Architecture map:
[`plan/compiler/architecture.md`](plan/compiler/architecture.md).
### `runtime/` — the C runtime (canonical, partially landed)
### `runtime/` — the C `wovm` VM
The root-level home of the C track. Today it holds the shipped thread-per-core io_uring reference (`wo-rt.c`, phases A–F done — see [`plan/exploration/c-runtime/00-plan.md`](plan/exploration/c-runtime/00-plan.md)) plus its `bench/` and Makefile. The VM and every runtime module land beside it under `src/`:
The register interpreter that loads and runs `.wob` images, plus the arena,
inferred GC, and the systems stdlib (`fs`/`time`/`env`/`net`/`proc`/`json`).
`runtime/src/main.c` also reads an image embedded in its own binary
(`/proc/self/exe`), which is how `woc build` produces a standalone executable.
`wo-rt.c` + `bench/` are the phase A–F io_uring event-loop reference that fed
the design (see [`plan/exploration/c-runtime/00-plan.md`](plan/exploration/c-runtime/00-plan.md)).
```
runtime/
├── Makefile wo-rt today; wovm + test targets as plan 1 lands
├── README.md orientation only (docs rule: real docs live under docs/)
├── wo-rt.c UNTOUCHED phase A–F event-loop reference
├── bench/ HTTP bench + goref/ Go comparison server
├── src/ ⏳ wob.h, obj, borrow, cont, gc, loader, vm, builtin, main (plan 1)
│ ⏳ sched, mailbox, shard (plan 4)
│ ⏳ table, wal, db (plan 5)
│ ⏳ http, json, router, handlers (plan 6)
│ ⏳ sys_env, sys_fs, sys_proc, sys_net, sys_time (plan 9)
└── test/ ⏳ t.h harness, wob_build assembler, test_*.c, cli_smoke.sh
```
Doctrine: C11, libc only, direct syscalls; computed-goto dispatch with
`-DWO_ISO_C` fallback; ASan/UBSan gates. Build: `just wovm-build`; gate:
`just wovm-test` (both dispatch flavors + `cli_smoke`).
Doctrine: C11, libc only, direct syscalls; computed-goto dispatch with `-DWO_ISO_C` fallback; ASan (+TSan from plan 4) gates.
### `database/` — the embedded engine
### `crates/` — the Rust runtime (active, retiring on parity)
Statically linked into every `wovm` (and every test binary) — one binary, no
separate database process. Class-shaped row slabs, an open-addressing id hash,
secondary-index multimaps with `@unique` enforcement, foreign-key restrict, and
a typed write-ahead log (`len|crc|payload|mark`, ack-after-fsync, torn-tail
drop, boot replay). Every class is a table; durability is opt-in via `WO_DATA`.
See `database/src/CODE-LOGIC.md`. The query surface that drives it lives in the
compiler (`emit.ml`), lowered to engine builtins — no SQL text in the image.
Unchanged from CLAUDE.md's description: `rt` is the monolithic Stage-2 runtime producing the `wo` binary; the 14 unprefixed siblings (`ql`, `value`, `engine`, `txn`, `db`, `wal`, `sub`, `http`, `gen`, `policy`, `logic`, `service`, `ui`, `app`) are phase scaffolds. **Retirement path:** stays authoritative until the C stack passes the parity harness (plan 3 task 7, plan 6 blog smoke); then moves under `.dev/reference/` the way v1 did. Until that day, nothing here is refactored to accommodate the C track.
### `tests/`, `scripts/`
### `tests/`, `scripts/` — the proof layer (target)
- `tests/corpus/` — the conformance spine (plan 3): `run/`, `compile-fail/`, `trap/`, then `gc/` (plan 3), `db/` (plan 5), `actor/` (plan 4), `sys/` (plan 9), `sample-logwatcher/` (plan 10), `lang/` (plan 8). Exact-outcome matching: byte-equal stdout, exact `WO-E###`, exact trap code.
- `scripts/` — `oop-e2e.sh` (corpus runner), `oop-parity.sh` (Rust-overlap manifest runner).
### `prototypes/` — reference implementations (frozen)
- `wo-db/` — the ~2k-line C++ query-layer prototype (SQL + Cypher + document paths). Stays: it is the semantic reference plan 5 cites. `make test` must keep passing.
- `wo-rt-c/` — ⚠ **stale duplicate.** The content moved to root `runtime/` (both are currently git-tracked). Slated for deletion once the user confirms nothing references it; `justfile`/doc references already point at `runtime/`. Nothing new lands here — repo rule: prototypes receive nothing new.
### `.dev/reference/` — read-only study material
- `crates/` — the 13 v1 `wo-*` crates, a nested Cargo workspace (build with `cd .dev/reference/crates && cargo build`). Preserved per the wo-seg migration plan ([`runtime/database/07-wo-seg-migration.md`](runtime/database/07-wo-seg-migration.md)).
- `colibri/`, `llama-cpp/` — vendored study trees (zero-dep C inference engine; MoE runtime) — see [`.dev/reference/colibri/`](.dev/reference/colibri/).
- `linux/`, `go/` — per-developer symlinks to kernel and Go sources (gitignored; recreate per `CLAUDE.md`).
- `rest/` — `.rest` HTTP files driving manual smoke against a running runtime; plan 6's blog smoke scripts the same sequences.
- `tests/corpus/` — the conformance spine: `run/`, `compile-fail/`, `trap/`,
`gc/`. Exact-outcome matching: byte-equal stdout, exact `WO-E###`, exact trap
code. Driven by `scripts/oop-e2e.sh` (`just oop-e2e`).
- `scripts/` — `oop-e2e.sh` (corpus), `mkdist.sh` + `install-accept.sh`
(tarball packaging), and the per-sample acceptance scripts
(`employee-accept.sh`, `log-watcher-accept.sh`).
### `docs/` — all documentation
```
docs/
├── 00-*,01,08-*.md status / principles / problem / structure docs
├── runtime/ the 7-phase database design series + runtime concept refs
├── examples/ log-watcher/, employee/, employee-list/ samples
├── plan/ numbered engineering plans 00–16, linux/ cards, assembly/,
│ ├── exploration/ c-runtime/ (A–F, done), linux/, postgresql/, assembly/
│ └── oop-vm/ ⏳ the OOP-track contracts: 00-wob-format, 01-error-catalog,
│ 02-corpus, 03-shard-actor, 04-db-binding, 05-http-service,
│ 07-systems-stdlib
├── superpowers/
│ ├── specs/ the two approved track specs (2026-08-01)
│ └── plans/ implementation plans 1–10 (2026-08-01, prose-only)
└── cm.md legacy notes
├── 00-*, 01-problem.md, 08-*.md status / principles / code-review / problem / structure
├── stories/ the canonical iteration arc (language-runtime-database/)
├── examples/ log-watcher/, employee/, employee-list/ samples
├── plan/ compiler/ plans, oop-vm/ contracts, exploration/ studies,
│ discarded.md + learnings.md registers
└── superpowers/ specs/ (approved designs) + plans/ (implementation plans)
```
Repo rule restated: documentation belongs here; code directories keep one orientation README each.
Repo rule: documentation belongs here; code directories keep one orientation
README each.
### `.dev/` — developer-local state
### `.dev/` — developer-local + reference
Gitignored symlinks into per-machine AI-tooling state plus `commit.md`, the running commit-message draft (the user commits; agents only append drafts). See `.dev/README.md`.
Gitignored per-machine tooling state plus `reference/` study trees: the v1
`wo-*` crates workspace, `colibri`/`llama-cpp` vendored studies, and
`linux/`/`go/` source symlinks. Read-only; nothing here is built by the main
gates.
## Lifecycle: how the shape evolves
## Build & test flow
| Stage | What changes at the root |
| --- | --- |
| Today | `runtime/` holds wo-rt.c; `crates/rt` serves Stage 2; plans are paper. |
| After plans 1–3 (milestone 1) | `compiler/` + `runtime/src/` + `tests/corpus/` + `scripts/` exist; `woc build` emits single binaries; `prototypes/wo-rt-c` deleted. |
| After plans 4–6 | `runtime/src/` carries shard/db/http modules; the C stack serves a sample end to end. |
| After plans 8–10 | systems stdlib in `runtime/src/sys_*`; `docs/examples/log-watcher/` proves program mode. |
| Parity | `crates/` moves to `.dev/reference/crates-v2/` (naming decided then); the `wo` toolchain name transfers to the C stack; Cargo.toml shrinks accordingly. |
`just woc-build` + `just wovm-build` produce the two binaries; `just oop-accept`
runs the full milestone gate (compile-time budget, conformance corpus under
ASan, single-binary smoke, both unit gates). Sample acceptance:
`just employee` (database), `just log-watcher` (systems stdlib). Packaging:
`just dist` → `writeonce-<ver>-linux-amd64.tar.gz`, proven by
`just install-accept`.
## Build sequence
## Naming conventions
The order the project completes, with the two parallel windows made explicit:
1. **`runtime/` VM core** — plan 1 (`.wob` format + `wovm`: memory model, loader, interpreter). No dependencies; the format doc it pins is everyone's contract.
2. **`compiler/` front** — plan 2 (`woc`: lexer→parser→types→owner). Independent of plan 1 — *may run in parallel with it*; needs `apt install ocaml dune`.
3. **Emit + end-to-end** — plan 3 (bytecode emitter, conformance corpus, `woc build` single binary, acceptance gate). Needs 1 + 2. **Milestone 1 done here.**
4. Two tracks fork and *can proceed in parallel*:
- **Server track (sequential within):** plan 4 shard-actor runtime → plan 5 DB engine binding → plan 6 HTTP service layer.
- **Systems track (sequential within):** plan 8 Haxe-parity language → plan 9 program mode + stdlib → plan 10 log-watcher sample. Only seam with the server track: the JSON codec shared between plans 6 and 9 (either lands it, noted in both).
5. **Parity + retirement** — parity harnesses green (plans 3/6), then `crates/` (Rust) retires to `.dev/reference/` and the `wo` name transfers to the C toolchain.
Rule of thumb: `runtime → compiler → emit → {shard → db → http} ∥ {language → stdlib → sample} → parity`.
## Naming conventions (recap)
- New runtime crates: unprefixed (`ql`, `db`, …). V1 crates: `wo-` prefixed, in `.dev/reference/crates/`.
- Binaries: `wo` (Rust toolchain today), `woc` (OCaml compiler), `wovm` (C VM); at parity the `wo` name moves to the C toolchain.
- C prototype directory names keep their historical `wo-` prefixes (`wo-db`).
- Plan/spec files: `YYYY-MM-DD-<topic>.md` under `docs/superpowers/{specs,plans}/`; numbered engineering plans under `docs/plan/`.
- Binaries: `woc` (OCaml compiler), `wovm` (C VM); a `woc build` / `woc <dir>`
output is named by the project's `wo.toml`.
- Plan/spec files: `YYYY-MM-DD-<topic>.md` under `docs/superpowers/{specs,plans}/`;
compiler plans under `docs/plan/compiler/`; normative contracts under
`docs/plan/oop-vm/`.
- Sample projects live under `docs/examples/<name>/` with their own `wo.toml`
and module `justfile`.

View file

@ -53,4 +53,5 @@ Status board: [`00-status.md`](../00-status.md) · Doctrine: [`../00-principles.
| **Minimal 3-file log-watcher sample** | Breaks the file-for-file `.hx` → `.wo` mapping and leaves the "could not express" column unproven — which is the sample's entire acceptance criterion. |
| **Raw code in plan documents** | Plans carry concept, reason, and required behavior in words; the executor writes the code. |
| **`##ui` / `.htmlx` LiveView frontend track** | 2026-08-17: removed the 9-doc `exploration/ui/` design set, the `14-mvc-ui-implementation` plan, and the `ui-htmlx-live` plan. All were built on the non-advancing Rust runtime (`.dev/reference/crates/wo-htmlx`, `cargo run`, WebSocket live-patches) and contradict the current woc/wovm direction. The 13d pricing-UI row went with them. Revisit only if a UI story is re-opened on the woc/wovm stack. |
| **Old-runtime "front door" + v1 design docs** | 2026-08-17: removed `writeonce-pl.md`, `runtime/wo-language.md`, `future-scope/ai-agents-content-management.md`, the numbered v1 set `02-recovery`/`03-data`/`04-ui`/`05-datalayer`/`06-markdown-render`/`07-ssl`, and `runtime/database/05-go-sdk.md`. They pitched the old Rust `wo` runtime (REST + LiveView + SQL/Cypher) as the current language and contradicted the shipped woc/wovm toolchain. The `runtime/database/` design series is kept as cited design history; the Rust-track plans/`done` are kept per the status board. |
| **Old-runtime "front door" + v1 design docs** | 2026-08-17: removed `writeonce-pl.md`, `runtime/wo-language.md`, `future-scope/ai-agents-content-management.md`, the numbered v1 set `02-recovery`/`03-data`/`04-ui`/`05-datalayer`/`06-markdown-render`/`07-ssl`, and `runtime/database/05-go-sdk.md`. They pitched the old Rust `wo` runtime (REST + LiveView + SQL/Cypher) as the current language and contradicted the shipped woc/wovm toolchain. |
| **The entire Rust `wo` runtime track** | 2026-08-18: removed `crates/` (the Stage-2 Rust runtime), `Cargo.toml`/`Cargo.lock`, `prototypes/` (wo-rt-c stale duplicate + wo-db C++ ref), the `rt-c-*` justfile recipes, the Rust engineering plans (`docs/plan/05..16`, `docs/plan/done/`), and `docs/runtime/` (the old runtime overview + 7-phase DB design series + async/fibers/gc/surreal essays). It was the prior, abandoned architecture — fully independent of the woc/wovm stack. Master now reflects only the current single-language project; the removed track lives in git history if ever needed as reference. Kept: the syscall/postgres/assembly/c-runtime **exploration studies** (they fed the current C runtime) and the discarded/learnings registers. |