shoney/writeonce
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

remove scrap

Browse source
This commit is contained in:
shoney.arickathil 2026-08-08 05:01:33 +02:00
parent 6c06df9255
commit 7a23bf6152
278 changed files with 35 additions and 14188 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

55
.gitignore vendored
View file

@ -1,15 +1,21 @@
# Cargo build artifacts # Cargo build artifacts
/target /target
/reference/crates/target /.dev/reference/crates/target
# C++ prototype build output # C++ prototype build output
/prototypes/*/build /prototypes/*/build
# C runtime prototype (prototypes/wo-rt-c): compiled binary + WAL/snapshot data. # OCaml/dune build artifacts (compiler/, the woc compiler front)
/compiler/_build
# C runtime (runtime/): compiled binary + WAL/snapshot data.
# `just rt-c-demo` and the default WO_DATA write ./wo-data at the repo root. # `just rt-c-demo` and the default WO_DATA write ./wo-data at the repo root.
/prototypes/wo-rt-c/wo-rt /runtime/wo-rt
/wo-data /wo-data
/prototypes/wo-rt-c/wo-data /runtime/wo-data
# wovm VM core: binary + test build dir (ASan test binaries, smoke fixtures)
/runtime/wovm
/runtime/build/
# Runtime data directories for the sample projects. # Runtime data directories for the sample projects.
# `wo.toml` points at `./data` which holds the per-project engine state. # `wo.toml` points at `./data` which holds the per-project engine state.
@ -19,23 +25,33 @@
# Legacy blog content and data (v1 writeonce storage) # Legacy blog content and data (v1 writeonce storage)
/content /content
# Symlinks to research source trees — user-specific absolute paths.
# Each contributor sets their own via:
# ln -s <path-to-linux-src> reference/linux
# ln -s <path-to-go-src> reference/go
# ln -s <path-to-postgresql-src> reference/postgresql
# ln -s <path-to-mcp-python-sdk-src> reference/mcp-python-sdk
# (clone: https://github.com/modelcontextprotocol/python-sdk.git)
/reference/linux
/reference/go
/reference/postgresql
/reference/mcp-python-sdk
# Developer-local links into agent-tooling state (Claude Code project data, # Developer-local links into agent-tooling state (Claude Code project data,
# opencode config/data, llama.cpp cache) — user-specific absolute paths. # opencode config/data, llama.cpp cache) — user-specific absolute paths.
# Only .dev/README.md is committed; recreate the links per machine as it describes. # Committed inside .dev: README.md and reference/ (v1 archive + rest smoke files);
# recreate the per-machine links as .dev/README.md describes.
/.dev/* /.dev/*
!/.dev/README.md !/.dev/README.md
!/.dev/reference/
# Symlinks to research source trees inside .dev/reference — user-specific
# absolute paths. Each contributor sets their own via:
# ln -s <path-to-linux-src> .dev/reference/linux
# ln -s <path-to-go-src> .dev/reference/go
# ln -s <path-to-postgresql-src> .dev/reference/postgresql
# ln -s <path-to-mcp-python-sdk-src> .dev/reference/mcp-python-sdk
# (clone: https://github.com/modelcontextprotocol/python-sdk.git)
# ln -s <path-to-colibri> .dev/reference/colibri
# ln -s <path-to-llama.cpp> .dev/reference/llama-cpp
/.dev/reference/linux
/.dev/reference/go
/.dev/reference/postgresql
/.dev/reference/mcp-python-sdk
/.dev/reference/colibri
/.dev/reference/llama-cpp
/.dev/reference/*
# Agent-orchestration scratch (SDD ledgers, briefs, review packages)
/.superpowers/
# Editor / OS noise — left broad on purpose so a contributor doesn't # Editor / OS noise — left broad on purpose so a contributor doesn't
# accidentally commit their IDE scratch or macOS metadata. # accidentally commit their IDE scratch or macOS metadata.
@ -47,7 +63,6 @@
!/.vscode/settings.json.example !/.vscode/settings.json.example
!/.vscode/extensions.json !/.vscode/extensions.json
/prototypes/wo-db
# Phase-F bench binaries (sources are committed; builds are not) # Phase-F bench binaries (sources are committed; builds are not)
/prototypes/wo-rt-c/bench/bench /runtime/bench/bench
/prototypes/wo-rt-c/bench/goref/goref /runtime/bench/goref/goref

9
crates/app/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "app"
version = "0.1.0"
edition = "2021"
description = "writeonce app manifest — `##app` routes, startup hooks, theme, i18n"
[lib]
name = "app"
path = "src/lib.rs"

20
crates/app/src/lib.rs
View file

@ -1,20 +0,0 @@
//! `app` — `##app` manifest: routes, theme, i18n, startup hooks.
//!
//! **Status: placeholder.** Phase 6 (see
//! [06-lowcode-fullstack.md § `##app`](
//! ../../../docs/runtime/database/06-lowcode-fullstack.md)).
//!
//! One `##app` block per project. Declares:
//! * the static route table (URL path → `ui.<screen>` binding, possibly
//! parameterised by dynamic segments like `/article/:slug`)
//! * cross-entity policies ("Admin bypasses row-level filters on every type")
//! * `on startup do …` hooks — idempotent seed code that runs once before
//! [`http`](../http/index.html) binds a listening socket
//! * theme tokens, i18n locale set, project metadata
//!
//! Consumed by [`ui`](../ui/index.html) for route rendering and by
//! [`policy`](../policy/index.html) for the cross-entity rules block.
//!
//! The [`docs/examples/blog/app.wo`](../../../docs/examples/blog/app.wo) and
//! [`docs/examples/ecommerce/app.wo`](../../../docs/examples/ecommerce/app.wo)
//! files are the reference shapes.

9
crates/db/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "db"
version = "0.1.0"
edition = "2021"
description = "writeonce top-level database facade — `open()`, `Tx`, `Query`, `Subscribe` over the engine"
[lib]
name = "db"
path = "src/lib.rs"

27
crates/db/src/lib.rs
View file

@ -1,27 +0,0 @@
//! `wo-db` — the Rust-facing database SDK.
//!
//! **Status: placeholder.** Phases 2–4 integrator.
//!
//! The top-level crate that the rest of the workspace (`wo-rt`, `wo-http`,
//! future `wo-serve` integration glue) depends on. Composes:
//!
//! | | |
//! | --- | --- |
//! | [`ql`](../ql/index.html) | parse |
//! | [`value`](../value/index.html) | row representation |
//! | [`engine`](../engine/index.html) | storage + execution |
//! | [`txn`](../txn/index.html) | transaction coordinator |
//! | [`wal`](../wal/index.html) | durability (Phase 3) |
//! | [`sub`](../sub/index.html) | live subscriptions (Phase 4) |
//!
//! Public surface the rest of the workspace will call:
//!
//! ```text
//! let db = db::open(&catalog, &options)?;
//! db.wo(ctx, ".wo source", params).await?; // raw DML
//! db.tx(ctx, |tx| async move { ... }).await?;
//! let sub = db.subscribe(ctx, "LIVE SELECT ...", params).await?;
//! ```
//!
//! See [05-go-sdk.md](../../../docs/runtime/database/05-go-sdk.md) — the Go
//! SDK's shape mirrors this one; both speak the same `.wo` wire protocol.

9
crates/engine/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "engine"
version = "0.1.0"
edition = "2021"
description = "writeonce in-memory executor — relational, document, and graph storage over a shared catalog"
[lib]
name = "engine"
path = "src/lib.rs"

19
crates/engine/src/lib.rs
View file

@ -1,19 +0,0 @@
//! `wo-engine` — the in-memory executor.
//!
//! **Status: placeholder.** Phase 2 (see
//! [02-wo-language.md](../../../docs/runtime/database/02-wo-language.md))
//! designs the three storage paradigms; Phase 3
//! ([03-inmemory-engine.md](../../../docs/runtime/database/03-inmemory-engine.md))
//! details the RAM-primary layout, arenas, MVCC version chains.
//!
//! The engine owns:
//! * relational tables — per-type `BTreeMap<id, Row>` today, B+ tree when
//! sized appropriately
//! * document collections — shape-free `Value::Object` rows; LSM when Phase 3
//! activates
//! * graph nodes + edges — adjacency over the type catalog
//! * a schema catalog compiled from `wo-ql` output
//!
//! Stage 2 ships this as a module inside [`rt`](../rt/index.html).
//! It extracts here when `wo-wal` and `wo-txn` need to reach into the same
//! structures.

9
crates/gen/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "gen"
version = "0.1.0"
edition = "2021"
description = "writeonce codegen — `.wo` schema → typed Go / TypeScript / Rust / Python clients"
[lib]
name = "gen"
path = "src/lib.rs"

21
crates/gen/src/lib.rs
View file

@ -1,21 +0,0 @@
//! `wo-gen` — client-SDK codegen.
//!
//! **Status: placeholder.** Phase 5 (see
//! [05-go-sdk.md § Typed SDK via `.wo` Schema Codegen](
//! ../../../docs/runtime/database/05-go-sdk.md)).
//!
//! Reads a compiled catalog (the same output [`ql`](../ql/index.html) +
//! [`engine`](../engine/index.html) produce from a `.wo` source tree)
//! and emits typed client code:
//!
//! * **Go** — structs with `wo:"column"` tags, `*Client`,
//! `TypedSubscription[T]` generics over channels
//! * **TypeScript** — types + `fetch` + WebSocket subscriptions
//! * **Rust** — `#[derive(Deserialize)]` structs + `impl Stream<Item = Delta>`
//! * **Python** — dataclasses + `async for delta in sub`
//! * **OpenAPI / GraphQL SDL** — machine-generated contracts served by
//! [`http`](../http/index.html)
//!
//! Invoked as `wo gen sdk --lang go --out ./client` via the toolchain; a bin
//! target will be added when the crate has real behaviour. Until then this is
//! an empty library scaffold.

9
crates/http/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "http"
version = "0.1.0"
edition = "2021"
description = "writeonce wire protocol — native binary + GraphQL over WebSocket + REST dispatch"
[lib]
name = "http"
path = "src/lib.rs"

20
crates/http/src/lib.rs
View file

@ -1,20 +0,0 @@
//! `wo-http` — HTTP + WebSocket + native wire protocol for the `.wo` runtime.
//!
//! **Status: placeholder.** Phase 4 (see
//! [04-client-api.md](../../../docs/runtime/database/04-client-api.md)).
//!
//! Three protocol surfaces, one dispatch:
//! * **REST** — JSON in/out, one route per `service rest` operation.
//! * **GraphQL over WebSocket** — SDL auto-generated from the schema; query,
//! mutation, `subscription` all routed through the same planner.
//! * **Native binary** — typed wire codec for `wo-db` and the Go SDK.
//!
//! Authentication middleware (`WithAPIKey`, `WithJWT`, `WithMTLS`) lives in
//! this crate — cross-cutting over every protocol surface.
//!
//! Stage 2 ships a minimal axum-based REST server inside
//! [`rt::server`](../rt/server/index.html). It migrates here when
//! Phase 4 activates and GraphQL / native surfaces join REST.
//!
//! Not the v1 `reference/crates/wo-http/` — that was a from-scratch HTTP
//! parser for the old single-threaded event loop.

9
crates/logic/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "logic"
version = "0.1.0"
edition = "2021"
description = "writeonce trigger + stored-fn compiler — type-attached `on <event>` and cross-entity `##logic`"
[lib]
name = "logic"
path = "src/lib.rs"

19
crates/logic/src/lib.rs
View file

@ -1,19 +0,0 @@
//! `wo-logic` — triggers and stored-function interpreter.
//!
//! **Status: placeholder.** Phase 6 (see
//! [06-lowcode-fullstack.md § `##logic`](
//! ../../../docs/runtime/database/06-lowcode-fullstack.md)).
//!
//! Two sources of runtime code:
//! * **Type-attached triggers** — `on insert | update | delete [when <pred>]
//! do <action>` declared inside a `type` body. Fires inside the committing
//! transaction so timestamp stamps (`paid_at`, `shipped_at`) are atomic
//! with the state change, never observable half-way.
//! * **Standalone `##logic`** — cross-entity workflows like the ecommerce
//! on-order-placed hook that decrements inventory on every line item.
//! * **`fn name(args) in txn ... { ... }`** — transactional stored
//! functions (the `checkout` in `docs/examples/ecommerce/logic/`).
//!
//! All three compile to the same intermediate form that [`txn`](../txn/index.html)
//! invokes during commit (for triggers) or directly on the main loop (for
//! `fn` calls from REST/native wire).

9
crates/policy/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "policy"
version = "0.1.0"
edition = "2021"
description = "writeonce policy engine — RBAC + row-level rules compiled into planner rewrites"
[lib]
name = "policy"
path = "src/lib.rs"

25
crates/policy/src/lib.rs
View file

@ -1,25 +0,0 @@
//! `wo-policy` — RBAC + row-level policies.
//!
//! **Status: placeholder.** Phase 6 (see
//! [06-lowcode-fullstack.md](../../../docs/runtime/database/06-lowcode-fullstack.md)).
//!
//! Compiles type-attached `policy` blocks (and standalone `##policy` blocks
//! for cross-entity rules) into predicates that [`engine`](../engine/index.html)
//! AND-s into every query at plan time. The rule is enforced once, at the
//! planner, so no caller can bypass it — not the REST handler, not the
//! `wo-db` SDK, not an admin script calling `fn` directly.
//!
//! Shape of a compiled policy:
//!
//! ```text
//! for type Article:
//! read: published == true OR $session.role == Admin
//! OR author == $session.user
//! write: $session.role == Admin OR ($session.role == Author && author == $session.user)
//! delete: $session.role == Admin
//! ```
//!
//! The planner merges the read predicate as a `WHERE` conjunct, the write
//! predicate as an assertion inside UPDATE/INSERT codegen, and delete as a
//! DELETE guard. Same mechanism as Postgres RLS — just authored declaratively
//! in the `type` block instead of via SQL migrations.

9
crates/ql/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "ql"
version = "0.1.0"
edition = "2021"
description = "writeonce query language — lexer, parser, AST for the `.wo` grammar"
[lib]
name = "ql"
path = "src/lib.rs"

13
crates/ql/src/lib.rs
View file

@ -1,13 +0,0 @@
//! `wo-ql` — the `.wo` grammar: lexer, parser, AST.
//!
//! **Status: placeholder.** Phase 2 of the [`.wo` runtime design](
//! ../../../docs/runtime/database/02-wo-language.md).
//!
//! Stage 2 of the runtime (shipped) carries the lexer/parser/AST inside
//! [`rt`](../rt/index.html) as part of its monolithic cut. This crate
//! is where those modules move to when Stage 3+ needs the grammar from
//! multiple places — the HTTP raw-query handler, `wo-gen`, the server-side
//! `fn` interpreter.
//!
//! Reference implementation: the C++ prototype at
//! [`prototypes/wo-db/`](../../../prototypes/wo-db/).

9
crates/service/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "service"
version = "0.1.0"
edition = "2021"
description = "writeonce service dispatcher — type-attached `service rest/graphql/native` and standalone `##service` bundles"
[lib]
name = "service"
path = "src/lib.rs"

19
crates/service/src/lib.rs
View file

@ -1,19 +0,0 @@
//! `service` — endpoint registration and dispatch.
//!
//! **Status: placeholder.** Phase 6 (see
//! [06-lowcode-fullstack.md § `##service`](
//! ../../../docs/runtime/database/06-lowcode-fullstack.md)).
//!
//! Walks every `service rest "/api/..." expose ...` block declared on a type
//! (and every standalone `##service` bundle for multi-entity APIs) and binds
//! handlers on [`http`](../http/index.html)'s router. Each exposed operation
//! (`list` / `get` / `create` / `update` / `delete` / `subscribe` / `me`)
//! maps to a predetermined shape — declarative CRUD is the whole point.
//!
//! `fn` endpoints (`POST /api/fn/checkout`) are also dispatched from here —
//! the service declares the function-as-endpoint wiring, [`logic`](../logic/index.html)
//! executes.
//!
//! Stage 2 does this work inline inside [`rt::server`](../rt/server/index.html).
//! It extracts here when Phase 6 adds GraphQL + native-protocol surfaces that
//! share the same service-block source of truth.

9
crates/sub/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "sub"
version = "0.1.0"
edition = "2021"
description = "writeonce live subscriptions — LIVE query registry + delta frames on commit"
[lib]
name = "sub"
path = "src/lib.rs"

21
crates/sub/src/lib.rs
View file

@ -1,21 +0,0 @@
//! `wo-sub` — the live-subscription engine.
//!
//! **Status: placeholder.** Phase 4 (see
//! [04-client-api.md](../../../docs/runtime/database/04-client-api.md)).
//!
//! Responsibilities:
//! * per-connection subscription registry — predicate + bound parameters
//! * commit-path dispatch: each [`txn`](../txn/index.html) commit
//! walks the registry, matches predicates against the delta set, and
//! pushes `Insert` / `Update` / `Delete` frames into per-subscription
//! ring buffers
//! * back-pressure policy (drop-and-resync / coalesce / disconnect) —
//! mirrors the client-side policy in [`wo-db`](../db/index.html)
//! / the Go SDK
//!
//! Stage 2 stubs the `/api/<type>/live` endpoint to 501 in
//! [`rt::server`](../rt/server/index.html). Stage 3 wires this crate
//! behind that route and upgrades the connection to WebSocket.
//!
//! Not related to the v1 `reference/crates/wo-sub/` crate — that one did
//! polling-based diff delivery over SSE for the old flat-file blog.

9
crates/txn/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "txn"
version = "0.1.0"
edition = "2021"
description = "writeonce transaction coordinator — MVCC, snapshot isolation, `RETURNING` alias table"
[lib]
name = "txn"
path = "src/lib.rs"

19
crates/txn/src/lib.rs
View file

@ -1,19 +0,0 @@
//! `wo-txn` — the cross-paradigm transaction coordinator.
//!
//! **Status: placeholder.** Phase 2 milestone 3 (see
//! [02-wo-language.md § Cross-Paradigm Transaction Coordinator](
//! ../../../docs/runtime/database/02-wo-language.md)).
//!
//! Responsibilities:
//! * `txn_id` allocation + snapshot timestamps
//! * ordering commits across `wo-engine`'s three storage paradigms
//! * the transaction-scoped **`RETURNING` alias table** — the state that
//! lets `INSERT … RETURNING id AS oid` thread `$oid` into a later
//! `CREATE (u)-[:PURCHASED {order_id: $oid}]->(p)` in the same
//! `BEGIN … COMMIT` block
//! * driving the in-process 2PC between the three engines on commit
//!
//! Under the single-threaded event-loop design (see
//! [§ Concurrency Model](../../../docs/runtime/database/02-wo-language.md#concurrency-model))
//! this reduces to a sequential counter + a per-txn name table. When the
//! runtime grows past one core via sharding, this crate owns cross-shard 2PC.

9
crates/ui/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "ui"
version = "0.1.0"
edition = "2021"
description = "writeonce UI renderer — `##ui` screens compile to SSR HTML + thin client runtime"
[lib]
name = "ui"
path = "src/lib.rs"

24
crates/ui/src/lib.rs
View file

@ -1,24 +0,0 @@
//! `ui` — `##ui` screens → SSR HTML + client-side delta runtime.
//!
//! **Status: placeholder.** Phase 6 (see
//! [06-lowcode-fullstack.md § `##ui`](
//! ../../../docs/runtime/database/06-lowcode-fullstack.md)).
//!
//! Declarative screens compile to:
//! * a **server-rendered HTML** tree — a compact template + data-binding
//! form computed from the screen's `source:` projection and `columns:` /
//! `sections:` declarations
//! * a **client-side runtime** (~50 KB vanilla JS) — opens a WebSocket to
//! the engine's [`sub`](../sub/index.html) endpoint, binds incoming delta
//! frames to DOM fragments by row key, handles sort / filter / paginate
//! without refetching
//! * an **auto-admin fallback** — any declared `type` without an explicit
//! `##ui` gets a generic CRUD screen
//!
//! `live: true` on a screen auto-generates a matching `LIVE SELECT` that the
//! client runtime subscribes to — no hand-written subscription code on the UI
//! side.
//!
//! The [`docs/examples/blog/ui/`](../../../docs/examples/blog/ui/) and
//! [`docs/examples/ecommerce/ui/`](../../../docs/examples/ecommerce/ui/)
//! directories are the reference shapes this crate has to handle.

9
crates/value/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "value"
version = "0.1.0"
edition = "2021"
description = "writeonce tagged `Value` + dotted-path utilities"
[lib]
name = "value"
path = "src/lib.rs"

14
crates/value/src/lib.rs
View file

@ -1,14 +0,0 @@
//! `wo-value` — the runtime's tagged `Value` type plus dotted-path helpers.
//!
//! **Status: placeholder.** Phase 2 foundation.
//!
//! Carries the union of all scalar + document + array shapes the engines pass
//! around: `Null`, `Bool`, `Int`, `Str`, `Array`, `Object`. Plus `fetch_path`
//! / `assign_path` utilities that drive the dotted-access semantics shared by
//! SQL expressions, document `UPDATE SET meta.x.y = z`, and Cypher projections
//! (see [the two-layer language spec](
//! ../../../docs/runtime/database/02-wo-language.md)).
//!
//! Stage 2 keeps these inside [`rt`](../rt/index.html)'s `engine` module;
//! they extract here once a second consumer (the WAL serializer, the wire codec)
//! needs them.

9
crates/wal/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "wal"
version = "0.1.0"
edition = "2021"
description = "writeonce write-ahead log — io_uring + fsync + crash recovery"
[lib]
name = "wal"
path = "src/lib.rs"

17
crates/wal/src/lib.rs
View file

@ -1,17 +0,0 @@
//! `wo-wal` — the write-ahead log.
//!
//! **Status: placeholder.** Phase 3 (see
//! [03-inmemory-engine.md](../../../docs/runtime/database/03-inmemory-engine.md)).
//!
//! Responsibilities:
//! * append committed txn records to an on-disk ring buffer via `io_uring`
//! * issue `IORING_OP_FSYNC` with a linked SQE for durability
//! * batch concurrent commits into one fsync per tick (group commit)
//! * drive crash recovery by replaying the log from the last checkpoint
//!
//! Under the single-threaded event-loop design, the WAL is owned directly by
//! the main loop — no separate WAL-writer thread. `io_uring`'s kernel-owned
//! SQPOLL handler does the draining; userland just submits and parks on CQEs.
//!
//! This is the crate that turns the RAM-primary engine from "cache" into
//! "database": commits aren't acked until their WAL record is on the SSD.

19
prototypes/wo-rt-c/Makefile
View file

@ -1,19 +0,0 @@
CC ?= cc
CFLAGS ?= -O2 -Wall -Wextra -std=c11
LDFLAGS ?= -pthread
wo-rt: wo-rt.c
$(CC) $(CFLAGS) -o $@ $< $(LDFLAGS)
bench/bench: bench/bench.c
$(CC) $(CFLAGS) -o $@ $< $(LDFLAGS)
bench: bench/bench
run: wo-rt
./wo-rt
clean:
rm -f wo-rt bench/bench
.PHONY: bench run clean

78
prototypes/wo-rt-c/README.md
View file

@ -1,78 +0,0 @@
# `wo-rt-c` — the writeonce runtime environment, in C
A single-file C implementation of the **runtime layer** the writeonce language runs on — now at **phase E: a durable RAM database**. Writes follow the dual-write order — RAM apply, framed WAL record (`len|crc32|payload|COMMIT`) to a per-shard `fallocate`'d log, one group-commit `fdatasync` per loop tick, **HTTP ack only after the fsync completion**. Boot performs the **first load, hard drive → RAM**: each shard replays its snapshot + WAL tail into its arena slice in parallel before any accept arms; clean shutdown snapshots each slice and truncates the WAL; a `meta` file pins the shard count so a mismatched `WO_THREADS` refuses to boot. `./wo-rt wal-check <file>` validates a log offline. `WO_THREADS` pinned threads (default = online cores), each owning its own raw io_uring ring (`io_uring_setup` + mmap'd SQ/CQ rings + `io_uring_enter` — **no liburing**), its own `SO_REUSEPORT` listener (multishot accept), its own keep-alive connections, and its own slice of the one mlock'd mmap arena — shared-nothing, no locks. Steady state is one `io_uring_enter` syscall per loop tick. Thread 0 owns the `signalfd`; shutdown broadcasts through per-thread `eventfd`s, both watched via `POLL_ADD` SQEs. **Zero dependencies beyond libc + kernel uapi headers.** The kernel is the runtime.
This is the runtime-layer sibling of [`prototypes/wo-db/`](../wo-db/) (the C++ query-layer prototype): a reference card showing, with no abstraction in the way, exactly which kernel primitives the production Rust runtime (`crates/rt/`) drives through `libc`. Same role, different layer.
```
prototypes/wo-db/ C++ what the LANGUAGE executes (parser, engine, transactions)
prototypes/wo-rt-c/ C what the RUNTIME stands on (epoll, signalfd, sockets)
crates/rt/ Rust the product — both layers, libc only
```
## Build, run, poke
```bash
make # cc -O2 -Wall -Wextra -std=c11 -pthread — no libraries
./wo-rt # 127.0.0.1:8085 (WO_PORT=9000 WO_THREADS=4 ./wo-rt to override)
curl localhost:8085/ # {"runtime":"wo-rt-c","loop":"epoll-et","threads":4,
# "shard":2,"shard_requests":[68,36,44,53]}
curl -X POST localhost:8085/api/notes -d '{"title":"hello"}'
# {"id":2,"title":"hello","shard":1} ← ids interleave per shard
curl localhost:8085/api/notes # the connection's shard only — shared-nothing
# ctrl-C → signalfd on shard 0 → eventfd broadcast → all shards join
```
Each connection hashes to one shard for life (`SO_REUSEPORT` 4-tuple): a list may land on a different shard than the create that preceded it. That is the architecture, not a bug — cross-shard reads are a later phase / design decision (see the [architecture doc's improvements](../../docs/plan/exploration/c-runtime/01-architecture.md)).
Or from the repo root: `just rt-c-demo`.
## Module map
Every block in `wo-rt.c` corresponds one-to-one to a module of the Rust runtime, which in turn mirrors Go's netpoller — the same lineage the docs trace:
| `wo-rt.c` block | Rust (`crates/rt/src/`) | Go (`reference/go/src/runtime/`) | Kernel reference card |
| --- | --- | --- | --- |
| `main` event loop (`epoll_create1` / `epoll_wait`, `EPOLLET`) | `runtime/netpoll_epoll.rs` | `netpoll_epoll.go` | [`linux/01-epoll.md`](../../docs/plan/exploration/linux/01-epoll.md) |
| `sig_setup` (`sigprocmask` + `signalfd`) | `runtime/signalfd.rs` | signal mask handling | [`linux/04-signalfd.md`](../../docs/plan/exploration/linux/04-signalfd.md) |
| `listener_bind` (`SOCK_NONBLOCK`, `accept4`-to-EAGAIN) | `http/listener.rs` | `net.Listen` + accept loop | `socket(7)` |
| `conn_drive` (read-to-EAGAIN, one buffer per fd) | `http/connection.rs` | `conn.Read` loop | the edge-triggered contract |
| `notes[]` store | `engine.rs` (BTreeMaps) | — | [`03-inmemory-engine.md`](../../docs/runtime/database/03-inmemory-engine.md) |
## What it demonstrates
- **One thread owns each shard outright.** Accept, parse, store, respond — no locks, no worker pool, no connection migration. Scaling past one core is more shards ([`09-concurrency-scaleout.md`](../../docs/plan/09-concurrency-scaleout.md)), never shared mutable state. The single cross-thread touch is the relaxed-atomic stats counters on `/` — monotonic, never on the data path.
- **Edge-triggered discipline.** Every registration sets `EPOLLET`; every readiness event is drained to `EAGAIN` (the accept loop and the read loop both). Get this wrong and connections silently hang — the reason the Rust module documents the same contract at the top of `netpoll_epoll.rs`.
- **Signals as fd events.** `SIGINT`/`SIGTERM` are blocked, then read from a `signalfd` on the same epoll — no async-signal-unsafe handler, no self-pipe trick.
- **RAM is the read path.** `GET /api/notes` touches a C array. The production engine is the same idea with MVCC and a WAL behind it.
## Architecture and roadmap
Documentation lives under `docs/` (repo convention) — this README stays here as the directory's orientation page only:
- [`docs/plan/exploration/c-runtime/01-architecture.md`](../../docs/plan/exploration/c-runtime/01-architecture.md) — the runtime defined by tracing **one memory address** through user space, kernel space, and hardware under a million concurrent connections, plus seven improvement proposals (seqlock reads, registered buffers, zero-copy send, SQPOLL, …).
- [`docs/plan/exploration/c-runtime/00-plan.md`](../../docs/plan/exploration/c-runtime/00-plan.md) — the phase sequence: **A → B → C → D → E → F, all ✅ shipped.**
## Measured (phase F, 20-core Linux 6.14, tmpfs data dir, `just rt-c-bench`)
Same C bench client (`bench/bench.c`, keep-alive, only 2xx counted) against both servers:
| Benchmark | **wo-rt-c** (8 shards, durable WAL, io_uring + group commit) | **Go `net/http`** (go1.25.1, 20 cores, no durability) | **Rust `wo`** (release, 8 shards, durable WAL, io_uring group commit)¹ |
| --- | --- | --- | --- |
| `GET /healthz` | **859,033 req/s** · p50 71 µs · p99 159 µs | 336,444 req/s · p50 70 µs · p99 1,277 µs | 746,340 req/s · p50 73 µs · p99 186 µs |
| `GET /` (JSON) | **671,312 req/s** · p99 180 µs | — | 692,671 req/s · p99 167 µs |
| `POST` write (tmpfs) | **618,343 commits/s** — fsync-acked · p99 194 µs | 320,516 req/s — RAM only, no WAL · p99 1,581 µs | 330,285 commits/s — fsync-acked · p99 360 µs |
| `POST` write (real ext4/NVMe) | — | — | **27,014 commits/s group commit vs 5,765 per-commit (4.7×)** · p50 2.2 ms |
| 10,000 idle conns | 0 errors | 0 errors | 0 errors |
¹ All numbers measured on a clean box with the same client (earlier parasite-contaminated runs superseded). The Rust column's history is the architecture roadmap, measured: 09a global mutex (74.9k writes/s) → 09b sharded engine (+51%) → 09c per-shard WAL (~1% durability cost) → **keep-alive: reads ×3.4 to 770k/s, durable writes ×1.9 to 331k/s, p99 under 350 µs everywhere**. Rust now beats Go on both columns *while fsyncing every write*, and sits within ~10% of the C prototype on reads — converging exactly as the same-architecture argument predicted. The one remaining C advantage is **group commit on io_uring** (one batched fsync + one syscall per tick vs per-commit fsync over epoll), which is the next port. Bonus finding: with `/tmp` accidentally full, the C runtime **refused to ack non-durable writes under ENOSPC** — the durability guarantee holding in an unplanned failure mode.
wo-rt-c on 8 cores outpaces Go on 20 with ~8× tighter p99 (Go's GC shows there) — while fsyncing every write Go doesn't. Honest caveats: `net/http` does full general-purpose HTTP; our parser is minimal; .NET was not installed on the box. **ACID under load:** three crash rounds (`kill -9` mid-bench at ~2M commits) all showed WAL records ≥ acked; isolation probe: 300 concurrent commits → 300 distinct ids; torn-tail records drop whole by CRC.
The crash-under-load test **found and fixed two real bugs** the lighter phase-D test missed: an ack-before-fsync race (`conn_continue` armed the send in the same tick the commit was staged) and an fd-reuse ABA hazard in ack parking (fixed with per-connection generation stamps). That is what phase F is for.
- [`docs/plan/exploration/c-runtime/02-single-binary.md`](../../docs/plan/exploration/c-runtime/02-single-binary.md) — the end goal: how the `wo build` **single binary** runs on this runtime environment — the runtime kernel is statically linked into every writeonce app (Go model, nothing to install), with the catalog/routes/bytecode payload consumed at boot.
## Deliberate simplifications
Single-shot RECV re-armed per request (multishot recv + buffer rings are a phase-F improvement), one outstanding SQE per connection, fixed-size buffers, naive `"title"` extraction instead of a JSON parser, no `timerfd`. Requires kernel ≥ 5.19 (multishot accept). This file is for reading; `crates/rt` is for running writeonce.

201
prototypes/wo-rt-c/bench/bench.c
View file

@ -1,201 +0,0 @@
/*
* bench.c — load client for wo-rt-c (phase F). Zero deps beyond libc.
*
* T threads, each driving ONE keep-alive connection in a tight request/
* response loop (TCP_NODELAY, full-response framing via Content-Length).
* Every request is latency-stamped; the run prints req/s, p50, p99, errors.
*
* usage: ./bench <host> <port> <threads> <seconds> <path> [post-json]
* read: ./bench 127.0.0.1 8085 64 5 /api/notes
* write: ./bench 127.0.0.1 8085 64 5 /api/notes '{"title":"bench"}'
* idle: ./bench 127.0.0.1 8085 10000 0 /healthz # seconds=0: open conns,
* # one request each, hold, exit
*/
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <errno.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#include <sys/socket.h>
#include <time.h>
#include <unistd.h>
#define MAX_SAMPLES 400000 /* per thread; counting continues past it */
static char g_req[2048];
static int g_reqlen;
static struct sockaddr_in g_addr;
static long g_deadline_us; /* 0 = idle-connection mode */
static int g_hold_secs;
struct worker {
pthread_t tid;
long reqs, errs;
long ok2xx, non2xx; /* honest accounting: only 2xx is success */
long *lat; /* µs samples */
int nlat;
};
static long now_us(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec * 1000000L + ts.tv_nsec / 1000;
}
static int conn_open(void) {
int fd = socket(AF_INET, SOCK_STREAM, 0);
if (fd < 0) return -1;
int one = 1;
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof one);
if (connect(fd, (struct sockaddr *)&g_addr, sizeof g_addr) < 0) { close(fd); return -1; }
return fd;
}
/* One request/response round-trip. Returns HTTP status, or -1 conn-dead. */
static int round_trip(int fd) {
size_t off = 0;
while (off < (size_t)g_reqlen) {
ssize_t n = write(fd, g_req + off, (size_t)g_reqlen - off);
if (n <= 0) { if (n < 0 && errno == EINTR) continue; return -1; }
off += (size_t)n;
}
static _Thread_local char buf[131072];
size_t got = 0, need = 0;
for (;;) {
ssize_t n = read(fd, buf + got, sizeof buf - 1 - got);
if (n <= 0) { if (n < 0 && errno == EINTR) continue; return -1; }
got += (size_t)n;
buf[got] = 0;
if (!need) {
char *he = strstr(buf, "\r\n\r\n");
if (!he) { if (got >= sizeof buf - 1) return -1; continue; }
size_t hdr = (size_t)(he + 4 - buf);
long cl = 0;
char *p = strcasestr(buf, "Content-Length:");
if (p) cl = strtol(p + 15, NULL, 10);
need = hdr + (size_t)cl;
}
if (got >= need) {
int code = 0;
sscanf(buf, "HTTP/1.1 %d", &code);
return code;
}
if (got >= sizeof buf - 1) return -1;
}
}
static void *worker_main(void *arg) {
struct worker *w = arg;
int fd = conn_open();
if (fd < 0) { w->errs++; return NULL; }
if (g_deadline_us == 0) { /* idle-connection mode */
if (round_trip(fd) > 0) w->reqs++; else w->errs++;
sleep((unsigned)g_hold_secs);
close(fd);
return NULL;
}
while (now_us() < g_deadline_us) {
long t0 = now_us();
int code = round_trip(fd);
if (code < 0) { /* reconnect once, then count errs */
close(fd);
fd = conn_open();
if (fd < 0) { w->errs++; break; }
w->errs++;
continue;
}
long dt = now_us() - t0;
if (w->nlat < MAX_SAMPLES) w->lat[w->nlat++] = dt;
w->reqs++;
if (code >= 200 && code < 300) w->ok2xx++; else w->non2xx++;
}
close(fd);
return NULL;
}
static int cmp_long(const void *a, const void *b) {
long x = *(const long *)a, y = *(const long *)b;
return (x > y) - (x < y);
}
int main(int argc, char **argv) {
if (argc < 6) {
fprintf(stderr, "usage: %s <host> <port> <threads> <seconds> <path> [post-json]\n", argv[0]);
return 2;
}
const char *host = argv[1];
int port = atoi(argv[2]);
int threads = atoi(argv[3]);
int secs = atoi(argv[4]);
const char *path = argv[5];
const char *body = argc > 6 ? argv[6] : NULL;
struct rlimit rl;
if (getrlimit(RLIMIT_NOFILE, &rl) == 0 && rl.rlim_cur < rl.rlim_max) {
rl.rlim_cur = rl.rlim_max;
setrlimit(RLIMIT_NOFILE, &rl);
}
memset(&g_addr, 0, sizeof g_addr);
g_addr.sin_family = AF_INET;
g_addr.sin_port = htons((uint16_t)port);
inet_pton(AF_INET, host, &g_addr.sin_addr);
if (body)
g_reqlen = snprintf(g_req, sizeof g_req,
"POST %s HTTP/1.1\r\nHost: %s\r\nContent-Type: application/json\r\n"
"Content-Length: %zu\r\nConnection: keep-alive\r\n\r\n%s",
path, host, strlen(body), body);
else
g_reqlen = snprintf(g_req, sizeof g_req,
"GET %s HTTP/1.1\r\nHost: %s\r\nConnection: keep-alive\r\n\r\n", path, host);
g_hold_secs = 3;
g_deadline_us = secs > 0 ? now_us() + (long)secs * 1000000L : 0;
struct worker *ws = calloc((size_t)threads, sizeof *ws);
for (int i = 0; i < threads; i++) {
ws[i].lat = secs > 0 ? malloc(MAX_SAMPLES * sizeof(long)) : NULL;
pthread_create(&ws[i].tid, NULL, worker_main, &ws[i]);
}
long t0 = now_us();
long total = 0, errs = 0, nlat = 0, ok = 0, bad = 0;
for (int i = 0; i < threads; i++) {
pthread_join(ws[i].tid, NULL);
total += ws[i].reqs;
errs += ws[i].errs;
nlat += ws[i].nlat;
ok += ws[i].ok2xx;
bad += ws[i].non2xx;
}
long wall_us = now_us() - t0;
if (secs == 0) {
printf("idle-conns: opened %ld / %d connections (errs %ld), held %ds, server survived\n",
total, threads, errs, g_hold_secs);
return errs ? 1 : 0;
}
long *all = malloc((size_t)nlat * sizeof(long));
long k = 0;
for (int i = 0; i < threads; i++) {
memcpy(all + k, ws[i].lat, (size_t)ws[i].nlat * sizeof(long));
k += ws[i].nlat;
}
qsort(all, (size_t)nlat, sizeof(long), cmp_long);
double rps = (double)ok / ((double)wall_us / 1e6);
printf("%-22s %d conns %ds %ld ok (2xx) %.0f ok/s p50 %ld µs p99 %ld µs non-2xx %ld errs %ld\n",
body ? "WRITE (POST)" : path, threads, secs, ok, rps,
nlat ? all[nlat / 2] : 0, nlat ? all[(long)((double)nlat * 0.99)] : 0, bad, errs);
return 0;
}

3
prototypes/wo-rt-c/bench/goref/go.mod
View file

@ -1,3 +0,0 @@
module goref
go 1.25.1

69
prototypes/wo-rt-c/bench/goref/main.go
View file

@ -1,69 +0,0 @@
// goref — the Go net/http comparison server for the phase-F benchmark.
// Same endpoints and semantics as wo-rt-c's RAM read path: /healthz, /,
// GET/POST /api/notes against an in-memory store. Durability is NOT
// implemented here (Go side has no WAL), so only READ benchmarks are
// apples-to-apples; the POST comparison measures Go's non-durable path
// against wo-rt-c's fsync-backed path and is labeled as such.
//
// go build -o goref . && ./goref # :8095, GOMAXPROCS = all cores
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"sync"
)
type note struct {
ID int `json:"id"`
Title string `json:"title"`
}
var (
mu sync.RWMutex
notes []note
nextID = 1
)
func main() {
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "ok")
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, `{"runtime":"go-net-http"}`)
})
http.HandleFunc("/api/notes", func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
mu.RLock()
b, _ := json.Marshal(notes)
mu.RUnlock()
w.Header().Set("Content-Type", "application/json")
w.Write(b)
case http.MethodPost:
var in struct {
Title string `json:"title"`
}
if json.NewDecoder(r.Body).Decode(&in) != nil || in.Title == "" {
http.Error(w, `{"error":"expected {\"title\":\"...\"}"}`, http.StatusBadRequest)
return
}
mu.Lock()
n := note{ID: nextID, Title: in.Title}
nextID++
if len(notes) < 100000 {
notes = append(notes, n)
}
mu.Unlock()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(n)
default:
http.Error(w, "method", http.StatusMethodNotAllowed)
}
})
fmt.Println("[goref] listening on :8095")
http.ListenAndServe("127.0.0.1:8095", nil)
}

979
prototypes/wo-rt-c/wo-rt.c
View file

@ -1,979 +0,0 @@
/*
* wo-rt.c — the writeonce runtime environment, in C. Phase E: first load.
*
* Phases A–D: N pinned threads with raw io_uring loops, SO_REUSEPORT
* listeners, keep-alive connections, one mlock'd mmap arena, and durable
* commits (RAM apply → framed WAL record → per-tick group fdatasync → ack
* on the fsync CQE). Phase E closes the loop: at boot — BEFORE any accept
* is armed — each shard thread loads its snapshot and replays its WAL into
* its arena slice, in parallel, validating every frame's CRC + COMMIT
* trailer and truncating at the first torn record. Appends resume at the
* validated tail. A clean shutdown writes a per-shard snapshot
* (`shard-<t>.data`) and truncates the WAL; boot prefers snapshot + WAL
* tail. The data directory carries a `meta` file pinning the shard count —
* a restart with a different WO_THREADS refuses to start (resharding is
* plan 09f, not silent data loss). Zero deps beyond libc + kernel uapi.
*
* build: make run: ./wo-rt [WO_PORT=8085 WO_THREADS=4 WO_DATA=./wo-data ./wo-rt]
* poke: curl -X POST localhost:8085/api/notes -d '{"title":"hello"}' # acked after fsync
* wal: ./wo-rt wal-check wo-data/shard-0.wal # offline frame/CRC validation
*
* Phase map: docs/plan/exploration/c-runtime/00-plan.md (A ✅ threads,
* B ✅ arena, C ✅ io_uring, D ✅ WAL, E this file, F bench). One-address
* trace: 01-architecture.md. Single-binary end goal: 02-single-binary.md.
*
* Module map (C ↔ Rust ↔ kernel reference card):
* ring_init/ring_enter ↔ (plan 09 decision 4: per-thread ring) ↔ linux/07-io_uring.md
* arena_init ↔ (plan 10 storage foundations) ↔ linux/08-mmap.md
* wal_flush / OP_FSYNC ↔ (plan 11 WAL + 09c per-shard WAL) ↔ linux/12-pwrite-fsync.md, 09-fallocate.md
* sig/evfd via POLL_ADD↔ runtime/{signalfd,eventfd}.rs ↔ linux/04-signalfd.md, 02-eventfd.md
*
* Requires IORING_FEAT_SINGLE_MMAP (≥5.4) and multishot accept (≥5.19).
* Phase D opens WALs with O_TRUNC (fresh log each boot) — replay-on-boot and
* snapshots are phase E; the crash test inspects the WAL offline via
* `wal-check` BEFORE any restart. Simplifications: single-shot RECV re-armed
* per request, naive JSON extraction, fixed-size WAL payloads.
*/
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <linux/io_uring.h>
#include <netinet/in.h>
#include <poll.h>
#include <pthread.h>
#include <sched.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/eventfd.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <sys/signalfd.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <time.h>
#include <unistd.h>
#ifndef IORING_ACCEPT_MULTISHOT
#define IORING_ACCEPT_MULTISHOT (1U << 0)
#endif
#define MAX_THREADS 64
#ifndef MAX_FDS
#define MAX_FDS 16384 /* conn slots per shard, fd-indexed */
#endif
#define IN_CAP 8192
#define OUT_CAP 65536
#define PAGE 4096
#define HUGE_2M (2u * 1024 * 1024)
#ifndef SLOT_SIZE
#define SLOT_SIZE 256 /* -D overridable for scale runs (phase F) */
#endif
#ifndef SLOTS_PER_SHARD
#define SLOTS_PER_SHARD 256
#endif
#define RING_ENTRIES 1024
#define WAL_PREALLOC (4u * 1024 * 1024) /* fallocate per shard */
#define WAL_COMMIT 0xC0FFEE42u /* frame trailer magic */
#define WAL_BATCH_CAP 65536 /* staged bytes per group commit */
#define WAL_BATCH_CONNS 256 /* acks parked per batch */
/* ---------------------------------------------------------------- crc32 --
* Hand-rolled (poly 0xEDB88320), table built once at boot. Zero deps. */
static uint32_t crc_table[256];
static void crc32_init(void) {
for (uint32_t i = 0; i < 256; i++) {
uint32_t c = i;
for (int k = 0; k < 8; k++) c = (c & 1) ? 0xEDB88320u ^ (c >> 1) : c >> 1;
crc_table[i] = c;
}
}
static uint32_t crc32(const void *buf, size_t len) {
const uint8_t *p = buf;
uint32_t c = 0xFFFFFFFFu;
while (len--) c = crc_table[(c ^ *p++) & 0xFF] ^ (c >> 8);
return c ^ 0xFFFFFFFFu;
}
/* ------------------------------------------------------------ WAL frame --
* [u32 len][u32 crc(payload)][payload][u32 WAL_COMMIT]. A record replays
* whole or not at all: bad len, bad crc, or missing trailer = torn tail.
* The payload carries the (shard,slot) coordinates phase B made stable. */
struct wal_payload {
uint32_t op; /* 1 = insert note */
uint32_t slot;
int32_t id;
char title[128];
};
#define WAL_FRAME_BYTES (4 + 4 + sizeof(struct wal_payload) + 4)
/* ---------------------------------------------------------------- arena --
* Unchanged from phase B: [header page][shard 0: bitmap page + slots]... */
struct arena_hdr {
char magic[8];
uint32_t version;
uint32_t n_shards;
uint32_t slots_per_shard;
uint32_t slot_size;
};
struct slot_note { int32_t id; char title[128]; };
static uint8_t *arena;
static size_t arena_bytes, arena_map_bytes, slice_bytes, bitmap_bytes;
static int arena_huge = 0, arena_locked = 0;
static int arena_init(int n_shards) {
bitmap_bytes = ((size_t)SLOTS_PER_SHARD / 8 + PAGE - 1) & ~((size_t)PAGE - 1);
slice_bytes = bitmap_bytes + (size_t)SLOTS_PER_SHARD * SLOT_SIZE;
arena_bytes = PAGE + (size_t)n_shards * slice_bytes;
arena_map_bytes = (arena_bytes + HUGE_2M - 1) & ~((size_t)HUGE_2M - 1);
arena = mmap(NULL, arena_map_bytes, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_POPULATE, -1, 0);
if (arena != MAP_FAILED) {
arena_huge = 1;
} else {
arena_map_bytes = (arena_bytes + PAGE - 1) & ~((size_t)PAGE - 1);
arena = mmap(NULL, arena_map_bytes, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0);
if (arena == MAP_FAILED) { perror("mmap arena"); return -1; }
}
arena_locked = (mlock(arena, arena_map_bytes) == 0);
if (!arena_locked)
fprintf(stderr, "[wo-rt-c] warn: mlock refused (%s) — arena not pinned\n", strerror(errno));
struct arena_hdr *hdr = (struct arena_hdr *)arena;
memcpy(hdr->magic, "WORTC\0\0", 8);
hdr->version = 3; /* phase C */
hdr->n_shards = (uint32_t)n_shards;
hdr->slots_per_shard = SLOTS_PER_SHARD;
hdr->slot_size = SLOT_SIZE;
return 0;
}
static uint64_t *shard_bitmap(int t) { return (uint64_t *)(arena + PAGE + (size_t)t * slice_bytes); }
static uint8_t *shard_slots (int t) { return arena + PAGE + (size_t)t * slice_bytes + bitmap_bytes; }
static struct slot_note *slot_at(int t, uint32_t i) {
return (struct slot_note *)(shard_slots(t) + (size_t)i * SLOT_SIZE);
}
/* ------------------------------------------------------------- io_uring --
* The raw ring: three pieces of memory shared with the kernel — the SQ/CQ
* ring headers+arrays (one mmap, IORING_FEAT_SINGLE_MMAP) and the SQE array.
* Submission: fill sqes[tail&mask], publish tail with a release store, tell
* the kernel with ONE io_uring_enter that also waits for completions. */
struct ring {
int fd;
unsigned *sq_head, *sq_tail, *sq_mask, *sq_array;
unsigned *cq_head, *cq_tail, *cq_mask;
struct io_uring_sqe *sqes;
struct io_uring_cqe *cqes;
unsigned local_tail; /* SQEs filled, not yet published */
unsigned to_submit;
};
static int ring_init(struct ring *r) {
struct io_uring_params p;
memset(&p, 0, sizeof p);
r->fd = (int)syscall(__NR_io_uring_setup, RING_ENTRIES, &p);
if (r->fd < 0) { perror("io_uring_setup"); return -1; }
if (!(p.features & IORING_FEAT_SINGLE_MMAP)) {
fprintf(stderr, "[wo-rt-c] kernel lacks IORING_FEAT_SINGLE_MMAP (need >= 5.4)\n");
return -1;
}
size_t sq_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned);
size_t cq_sz = p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe);
size_t sz = sq_sz > cq_sz ? sq_sz : cq_sz;
uint8_t *sqcq = mmap(NULL, sz, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE,
r->fd, IORING_OFF_SQ_RING);
if (sqcq == MAP_FAILED) { perror("mmap sq/cq ring"); return -1; }
r->sq_head = (unsigned *)(sqcq + p.sq_off.head);
r->sq_tail = (unsigned *)(sqcq + p.sq_off.tail);
r->sq_mask = (unsigned *)(sqcq + p.sq_off.ring_mask);
r->sq_array = (unsigned *)(sqcq + p.sq_off.array);
r->cq_head = (unsigned *)(sqcq + p.cq_off.head);
r->cq_tail = (unsigned *)(sqcq + p.cq_off.tail);
r->cq_mask = (unsigned *)(sqcq + p.cq_off.ring_mask);
r->cqes = (struct io_uring_cqe *)(sqcq + p.cq_off.cqes);
r->sqes = mmap(NULL, p.sq_entries * sizeof(struct io_uring_sqe),
PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE,
r->fd, IORING_OFF_SQES);
if (r->sqes == MAP_FAILED) { perror("mmap sqes"); return -1; }
r->local_tail = *r->sq_tail;
r->to_submit = 0;
return 0;
}
static struct io_uring_sqe *sqe_get(struct ring *r) {
unsigned idx = r->local_tail & *r->sq_mask;
struct io_uring_sqe *s = &r->sqes[idx];
memset(s, 0, sizeof *s);
r->sq_array[idx] = idx;
r->local_tail++;
r->to_submit++;
return s;
}
/* user_data = (op << 32) | fd-or-batch-index */
enum { OP_ACCEPT = 1, OP_RECV, OP_SEND, OP_EVFD, OP_SIGFD, OP_WALWR, OP_FSYNC };
static uint64_t ud(int op, int fd) { return ((uint64_t)op << 32) | (uint32_t)fd; }
static int ring_enter(struct ring *r, unsigned wait) {
__atomic_store_n(r->sq_tail, r->local_tail, __ATOMIC_RELEASE);
unsigned n = r->to_submit;
r->to_submit = 0;
for (;;) {
int rc = (int)syscall(__NR_io_uring_enter, r->fd, n, wait,
IORING_ENTER_GETEVENTS, NULL, 0);
if (rc >= 0) return rc;
if (errno == EINTR) { n = 0; continue; } /* already submitted */
perror("io_uring_enter");
return -1;
}
}
/* ----------------------------------------------------------- connection --
* Keep-alive state machine. Exactly one outstanding SQE per connection:
* RECV while a request is being assembled, SEND while a response drains.
* Leftover bytes after a request (pipelining) are carried over and parsed
* before the next RECV is armed. */
struct conn {
char in[IN_CAP]; size_t in_len;
char out[OUT_CAP]; size_t out_len, out_off;
int in_use;
int closing; /* close once the out buffer drains */
int await_durable;/* response parked until this tick's fsync CQE */
uint64_t gen; /* incarnation stamp — kernel fds get reused */
};
/* One group-commit batch: staged WAL bytes + the connections whose acks ride
* its fsync. Double-buffered: while batch[k] is in flight (write→fsync
* linked SQEs), new commits stage into batch[k^1].
* Acks are parked as (fd, gen) pairs: an fd alone is ABA-unsafe — a parked
* connection can die, the kernel reuses its fd for a NEW connection whose
* commit sits in the OTHER batch, and a bare-fd release would ack that new
* connection before ITS record is durable. Found by the phase-F crash test
* (7 acked-but-unwritten records out of ~990k under reconnect churn). */
struct wal_batch {
char buf[WAL_BATCH_CAP];
size_t len;
int conns[WAL_BATCH_CONNS];
uint64_t gens[WAL_BATCH_CONNS];
int n_conns;
};
struct shard {
int id;
int lfd, evfd;
struct ring ring;
pthread_t tid;
int next_id;
_Atomic int used;
int wal_fd;
_Atomic size_t wal_off; /* owner-written; stats-readable cross-shard */
struct wal_batch batch[2];
int active; /* batch being staged */
int in_flight; /* a write→fsync pair is on the ring */
char snap_path[320];
struct conn conns[MAX_FDS];
};
/* Snapshot file: [snap_hdr][bitmap page][slot bytes]. Written on clean
* shutdown, loaded at boot before WAL replay. */
struct snap_hdr {
char magic[8]; /* "WOSNAP\0\0" */
uint32_t version;
int32_t next_id;
int32_t used;
};
static struct shard *shards;
static int n_threads = 1;
static int sigfd = -1;
static _Atomic unsigned long reqs[MAX_THREADS];
static int slot_alloc(struct shard *sh) {
uint64_t *bm = shard_bitmap(sh->id);
for (uint32_t w = 0; w < SLOTS_PER_SHARD / 64; w++) {
if (bm[w] == UINT64_MAX) continue;
uint32_t b = (uint32_t)__builtin_ctzll(~bm[w]);
uint32_t i = w * 64 + b;
if (i >= SLOTS_PER_SHARD) break;
bm[w] |= (1ULL << b);
atomic_fetch_add_explicit(&sh->used, 1, memory_order_relaxed);
return (int)i;
}
return -1;
}
/* --------------------------------------------------------- SQE builders -- */
static void arm_accept(struct shard *sh) { /* multishot: arm once */
struct io_uring_sqe *s = sqe_get(&sh->ring);
s->opcode = IORING_OP_ACCEPT;
s->fd = sh->lfd;
s->ioprio = IORING_ACCEPT_MULTISHOT;
s->user_data = ud(OP_ACCEPT, sh->lfd);
}
static void arm_poll(struct shard *sh, int fd, int op) {
struct io_uring_sqe *s = sqe_get(&sh->ring);
s->opcode = IORING_OP_POLL_ADD;
s->fd = fd;
s->poll_events = POLLIN;
s->user_data = ud(op, fd);
}
static void arm_recv(struct shard *sh, int fd) {
struct conn *c = &sh->conns[fd];
struct io_uring_sqe *s = sqe_get(&sh->ring);
s->opcode = IORING_OP_RECV;
s->fd = fd;
s->addr = (uint64_t)(uintptr_t)(c->in + c->in_len);
s->len = (uint32_t)(IN_CAP - 1 - c->in_len);
s->user_data = ud(OP_RECV, fd);
}
static void arm_send(struct shard *sh, int fd) {
struct conn *c = &sh->conns[fd];
struct io_uring_sqe *s = sqe_get(&sh->ring);
s->opcode = IORING_OP_SEND;
s->fd = fd;
s->addr = (uint64_t)(uintptr_t)(c->out + c->out_off);
s->len = (uint32_t)(c->out_len - c->out_off);
s->msg_flags = MSG_NOSIGNAL;
s->user_data = ud(OP_SEND, fd);
}
/* ------------------------------------------------------------ WAL flush --
* Called once per loop tick. If commits were staged and no batch is in
* flight, submit ONE write SQE for the whole batch at the shard's tail
* offset, hard-linked to ONE fdatasync SQE. Every parked ack in the batch
* is released when the fsync CQE arrives — group commit. */
static void wal_flush(struct shard *sh) {
if (sh->in_flight) return;
struct wal_batch *b = &sh->batch[sh->active];
if (b->len == 0) return;
size_t off = atomic_load_explicit(&sh->wal_off, memory_order_relaxed);
struct io_uring_sqe *w = sqe_get(&sh->ring);
w->opcode = IORING_OP_WRITE;
w->fd = sh->wal_fd;
w->addr = (uint64_t)(uintptr_t)b->buf;
w->len = (uint32_t)b->len;
w->off = off;
w->flags = IOSQE_IO_LINK; /* fsync follows the write */
w->user_data = ud(OP_WALWR, sh->active);
struct io_uring_sqe *f = sqe_get(&sh->ring);
f->opcode = IORING_OP_FSYNC;
f->fd = sh->wal_fd;
f->fsync_flags = IORING_FSYNC_DATASYNC;
f->user_data = ud(OP_FSYNC, sh->active);
sh->in_flight = 1;
sh->active ^= 1; /* new commits stage in the twin */
}
/* Stage one commit's frame + park the connection's ack on the active batch.
* Returns 0 if the batch has no room (caller responds 503, no RAM apply). */
static int wal_append(struct shard *sh, int connfd, uint32_t slot, int32_t id, const char *title) {
struct wal_batch *b = &sh->batch[sh->active];
if (b->len + WAL_FRAME_BYTES > WAL_BATCH_CAP || b->n_conns >= WAL_BATCH_CONNS)
return 0;
struct wal_payload p;
memset(&p, 0, sizeof p);
p.op = 1;
p.slot = slot;
p.id = id;
snprintf(p.title, sizeof p.title, "%s", title);
b->gens[b->n_conns] = sh->conns[connfd].gen;
uint32_t len = (uint32_t)sizeof p;
uint32_t crc = crc32(&p, sizeof p);
uint32_t end = WAL_COMMIT;
char *dst = b->buf + b->len;
memcpy(dst, &len, 4);
memcpy(dst + 4, &crc, 4);
memcpy(dst + 8, &p, sizeof p);
memcpy(dst + 8 + sizeof p, &end, 4);
b->len += WAL_FRAME_BYTES;
b->conns[b->n_conns++] = connfd;
return 1;
}
/* ----------------------------------------------------------------- http -- */
static void respond(struct conn *c, const char *status, const char *ctype, const char *body) {
size_t blen = strlen(body);
int n = snprintf(c->out + c->out_len, OUT_CAP - c->out_len,
"HTTP/1.1 %s\r\nContent-Type: %s\r\nContent-Length: %zu\r\nConnection: %s\r\n\r\n%s",
status, ctype, blen, c->closing ? "close" : "keep-alive", body);
if (n > 0 && (size_t)n < OUT_CAP - c->out_len) c->out_len += (size_t)n;
else c->closing = 1; /* response too big — drop conn */
}
static int json_title(const char *body, char *out, size_t cap) {
const char *p = strstr(body, "\"title\"");
if (!p) return 0;
p = strchr(p + 7, ':'); if (!p) return 0;
p = strchr(p, '"'); if (!p) return 0;
p++;
size_t i = 0;
while (*p && *p != '"' && i + 1 < cap) out[i++] = *p++;
out[i] = 0;
return i > 0;
}
static void route(struct shard *sh, struct conn *c, const char *method, const char *path, const char *body) {
atomic_fetch_add_explicit(&reqs[sh->id], 1, memory_order_relaxed);
if (!strcmp(method, "GET") && !strcmp(path, "/")) {
char out[1024];
size_t off = (size_t)snprintf(out, sizeof out,
"{\"runtime\":\"wo-rt-c\",\"loop\":\"io_uring\",\"threads\":%d,\"shard\":%d,"
"\"arena\":{\"bytes\":%zu,\"mapped\":%zu,\"hugepages\":%s,\"mlocked\":%s,"
"\"slot_size\":%d,\"slots_per_shard\":%d},\"shard_used\":[",
n_threads, sh->id, arena_bytes, arena_map_bytes,
arena_huge ? "true" : "false", arena_locked ? "true" : "false",
SLOT_SIZE, SLOTS_PER_SHARD);
for (int t = 0; t < n_threads; t++)
off += (size_t)snprintf(out + off, sizeof out - off, "%s%d", t ? "," : "",
atomic_load_explicit(&shards[t].used, memory_order_relaxed));
off += (size_t)snprintf(out + off, sizeof out - off, "],\"shard_requests\":[");
for (int t = 0; t < n_threads; t++)
off += (size_t)snprintf(out + off, sizeof out - off, "%s%lu", t ? "," : "",
atomic_load_explicit(&reqs[t], memory_order_relaxed));
off += (size_t)snprintf(out + off, sizeof out - off, "],\"wal_bytes\":[");
for (int t = 0; t < n_threads; t++)
off += (size_t)snprintf(out + off, sizeof out - off, "%s%zu", t ? "," : "",
atomic_load_explicit(&shards[t].wal_off, memory_order_relaxed));
snprintf(out + off, sizeof out - off, "]}");
respond(c, "200 OK", "application/json", out);
} else if (!strcmp(method, "GET") && !strcmp(path, "/healthz")) {
respond(c, "200 OK", "text/plain", "ok");
} else if (!strcmp(method, "GET") && !strcmp(path, "/api/notes")) {
static _Thread_local char out[SLOTS_PER_SHARD * 160 + 64];
uint64_t *bm = shard_bitmap(sh->id);
size_t off = (size_t)snprintf(out, sizeof out, "{\"shard\":%d,\"notes\":[", sh->id);
int first = 1;
for (uint32_t i = 0; i < SLOTS_PER_SHARD; i++) {
if (!(bm[i / 64] & (1ULL << (i % 64)))) continue;
struct slot_note *n = slot_at(sh->id, i);
off += (size_t)snprintf(out + off, sizeof out - off,
"%s{\"id\":%d,\"title\":\"%s\"}", first ? "" : ",", n->id, n->title);
first = 0;
}
snprintf(out + off, sizeof out - off, "]}");
respond(c, "200 OK", "application/json", out);
} else if (!strcmp(method, "POST") && !strcmp(path, "/api/notes")) {
char title[128];
int i;
if (!json_title(body, title, sizeof title)) {
respond(c, "400 Bad Request", "application/json", "{\"error\":\"expected {\\\"title\\\":\\\"...\\\"}\"}");
return;
}
/* Capacity gates BEFORE the RAM apply — an aborted op writes nothing. */
struct wal_batch *b = &sh->batch[sh->active];
if (b->len + WAL_FRAME_BYTES > WAL_BATCH_CAP || b->n_conns >= WAL_BATCH_CONNS) {
respond(c, "503 Service Unavailable", "application/json", "{\"error\":\"commit batch full, retry\"}");
return;
}
if ((i = slot_alloc(sh)) < 0) {
respond(c, "507 Insufficient Storage", "application/json", "{\"error\":\"shard full\"}");
return;
}
struct slot_note *n = slot_at(sh->id, (uint32_t)i); /* 1. the RAM apply */
n->id = sh->next_id;
sh->next_id += n_threads;
snprintf(n->title, sizeof n->title, "%s", title);
char out[224];
snprintf(out, sizeof out, "{\"id\":%d,\"title\":\"%s\",\"shard\":%d,\"slot\":%d}",
n->id, n->title, sh->id, i);
respond(c, "201 Created", "application/json", out); /* built, NOT sent */
int connfd = (int)(c - sh->conns); /* conns is fd-indexed */
wal_append(sh, connfd, (uint32_t)i, n->id, n->title);/* 2. stage WAL frame */
c->await_durable = 1; /* 4. ack rides fsync */
} else {
respond(c, "404 Not Found", "application/json", "{\"error\":\"no such route\"}");
}
}
/* ----------------------------------------------------- state machine ----- */
static void conn_open(struct shard *sh, int fd) {
struct conn *c = &sh->conns[fd];
c->in_len = c->out_len = c->out_off = 0;
c->closing = 0;
c->await_durable = 0;
c->gen++; /* new incarnation — stale parked acks won't match */
c->in_use = 1;
}
static void conn_close(struct shard *sh, int fd) {
if (fd >= 0 && fd < MAX_FDS) sh->conns[fd].in_use = 0;
close(fd);
}
/* Try to consume ONE complete request from the in buffer. Returns 1 if a
* response was produced (out has bytes), 0 if the request is incomplete. */
static int try_process(struct shard *sh, struct conn *c) {
c->in[c->in_len] = 0;
char *hdr_end = strstr(c->in, "\r\n\r\n");
if (!hdr_end) return 0;
char *body = hdr_end + 4;
size_t total = (size_t)(body - c->in);
const char *cl = strcasestr(c->in, "Content-Length:");
if (cl) {
long want = strtol(cl + 15, NULL, 10);
if (want < 0) want = 0;
if (c->in_len < total + (size_t)want) return 0;
total += (size_t)want;
}
/* HTTP/1.1 defaults to keep-alive; honor an explicit close. */
if (strcasestr(c->in, "connection: close") ||
(strstr(c->in, "HTTP/1.0") && !strcasestr(c->in, "connection: keep-alive")))
c->closing = 1;
char method[8] = {0}, path[256] = {0};
if (sscanf(c->in, "%7s %255s", method, path) == 2)
route(sh, c, method, path, body);
else
c->closing = 1;
memmove(c->in, c->in + total, c->in_len - total); /* carry pipelined tail */
c->in_len -= total;
return 1;
}
/* Advance a connection: drain out via SEND, else parse, else arm RECV.
* A parked commit ack arms nothing — the fsync CQE handler resumes it. */
static void conn_continue(struct shard *sh, int fd) {
struct conn *c = &sh->conns[fd];
if (c->await_durable) { return; }
if (c->out_off < c->out_len) { arm_send(sh, fd); return; }
c->out_off = c->out_len = 0;
if (c->closing) { conn_close(sh, fd); return; }
if (try_process(sh, c)) {
/* route() may have JUST parked this response (await set inside
* try_process) — sending now would race the fsync. The fsync CQE
* re-enters here with await cleared and arms the send.
* (Found by the phase-F crash-under-load test: ~6 acked-but-
* unwritten records per ~750k at the kill instant.) */
if (!c->await_durable) arm_send(sh, fd);
return;
}
if (c->in_len >= IN_CAP - 1) { conn_close(sh, fd); return; } /* oversize head */
arm_recv(sh, fd);
}
/* --------------------------------------------------------------- recovery --
* First load: hard drive → RAM, per shard, in parallel, BEFORE accept arms.
* Snapshot (if any) restores the slice wholesale; the WAL tail replays
* commits since that snapshot. Frame validation is wal-check's logic with
* the printf swapped for the arena apply. */
static long now_ms(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec * 1000 + ts.tv_nsec / 1000000;
}
static int snap_load(struct shard *sh) {
int fd = open(sh->snap_path, O_RDONLY | O_CLOEXEC);
if (fd < 0) return 0;
struct snap_hdr h;
size_t slot_bytes = (size_t)SLOTS_PER_SHARD * SLOT_SIZE;
if (read(fd, &h, sizeof h) != (ssize_t)sizeof h ||
memcmp(h.magic, "WOSNAP\0", 8) != 0 ||
pread(fd, shard_bitmap(sh->id), bitmap_bytes, (off_t)sizeof h) != (ssize_t)bitmap_bytes ||
pread(fd, shard_slots(sh->id), slot_bytes, (off_t)(sizeof h + bitmap_bytes)) != (ssize_t)slot_bytes) {
fprintf(stderr, "[wo-rt-c] shard %d: snapshot unreadable — starting from WAL only\n", sh->id);
memset(shard_bitmap(sh->id), 0, bitmap_bytes + slot_bytes);
close(fd);
return 0;
}
sh->next_id = h.next_id;
atomic_store_explicit(&sh->used, h.used, memory_order_relaxed);
close(fd);
return h.used;
}
static int wal_replay(struct shard *sh) {
size_t off = 0;
int recs = 0;
int32_t maxid = 0;
for (;;) {
uint32_t len, crc, end;
struct wal_payload p;
if (pread(sh->wal_fd, &len, 4, (off_t)off) != 4 || len == 0) break;
if (len != sizeof p ||
pread(sh->wal_fd, &crc, 4, (off_t)(off + 4)) != 4 ||
pread(sh->wal_fd, &p, sizeof p, (off_t)(off + 8)) != (ssize_t)sizeof p ||
pread(sh->wal_fd, &end, 4, (off_t)(off + 8 + sizeof p)) != 4 ||
crc32(&p, sizeof p) != crc || end != WAL_COMMIT) {
fprintf(stderr, "[wo-rt-c] shard %d: torn WAL record at byte %zu — truncating\n",
sh->id, off);
break;
}
if (p.op == 1 && p.slot < SLOTS_PER_SHARD) { /* idempotent apply */
uint64_t *bm = shard_bitmap(sh->id);
if (!(bm[p.slot / 64] & (1ULL << (p.slot % 64)))) {
bm[p.slot / 64] |= (1ULL << (p.slot % 64));
atomic_fetch_add_explicit(&sh->used, 1, memory_order_relaxed);
}
struct slot_note *n = slot_at(sh->id, p.slot);
n->id = p.id;
snprintf(n->title, sizeof n->title, "%s", p.title);
if (p.id > maxid) maxid = p.id;
}
recs++;
off += WAL_FRAME_BYTES;
}
/* Resume appends at the validated tail; drop torn bytes, re-preallocate. */
if (ftruncate(sh->wal_fd, (off_t)off) == 0)
(void)!fallocate(sh->wal_fd, 0, 0, off > WAL_PREALLOC ? off : WAL_PREALLOC);
atomic_store_explicit(&sh->wal_off, off, memory_order_relaxed);
if (maxid > 0 && maxid + n_threads > sh->next_id)
sh->next_id = maxid + n_threads; /* interleaved high-water */
return recs;
}
/* Clean-shutdown snapshot: write slice → fsync → atomic rename → truncate WAL. */
static void snap_write(struct shard *sh) {
char tmp[336];
snprintf(tmp, sizeof tmp, "%s.tmp", sh->snap_path);
int fd = open(tmp, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
if (fd < 0) { perror("snapshot open"); return; }
struct snap_hdr h;
memset(&h, 0, sizeof h);
memcpy(h.magic, "WOSNAP\0", 8);
h.version = 1;
h.next_id = sh->next_id;
h.used = atomic_load_explicit(&sh->used, memory_order_relaxed);
size_t slot_bytes = (size_t)SLOTS_PER_SHARD * SLOT_SIZE;
int ok = write(fd, &h, sizeof h) == (ssize_t)sizeof h
&& write(fd, shard_bitmap(sh->id), bitmap_bytes) == (ssize_t)bitmap_bytes
&& write(fd, shard_slots(sh->id), slot_bytes) == (ssize_t)slot_bytes
&& fsync(fd) == 0;
close(fd);
if (!ok || rename(tmp, sh->snap_path) < 0) { fprintf(stderr, "[wo-rt-c] shard %d: snapshot failed\n", sh->id); unlink(tmp); return; }
if (ftruncate(sh->wal_fd, 0) == 0) { /* WAL now redundant */
(void)!fallocate(sh->wal_fd, 0, 0, WAL_PREALLOC);
fsync(sh->wal_fd);
}
printf("[wo-rt-c] shard %d: snapshot %d rows → %s, wal truncated\n", sh->id, h.used, sh->snap_path);
}
/* ------------------------------------------------------------ shard loop -- */
static void *shard_main(void *arg) {
struct shard *sh = arg;
struct ring *r = &sh->ring;
cpu_set_t set;
CPU_ZERO(&set);
CPU_SET((unsigned)sh->id % (unsigned)sysconf(_SC_NPROCESSORS_ONLN), &set);
pthread_setaffinity_np(pthread_self(), sizeof set, &set);
/* First load: disk → RAM, before any accept is armed. */
long t0 = now_ms();
int srows = snap_load(sh);
int wrecs = wal_replay(sh);
if (srows || wrecs)
printf("[wo-rt-c] shard %d: recovered %d snapshot rows + %d wal records in %ld ms\n",
sh->id, srows, wrecs, now_ms() - t0);
arm_accept(sh);
arm_poll(sh, sh->evfd, OP_EVFD);
if (sh->id == 0) arm_poll(sh, sigfd, OP_SIGFD);
for (;;) {
if (ring_enter(r, 1) < 0) break; /* ONE syscall per tick */
unsigned head = *r->cq_head;
unsigned tail = __atomic_load_n(r->cq_tail, __ATOMIC_ACQUIRE);
for (; head != tail; head++) {
struct io_uring_cqe *cqe = &r->cqes[head & *r->cq_mask];
int op = (int)(cqe->user_data >> 32);
int fd = (int)(uint32_t)cqe->user_data;
int res = cqe->res;
switch (op) {
case OP_EVFD: {
uint64_t v;
(void)!read(sh->evfd, &v, sizeof v);
__atomic_store_n(r->cq_head, head + 1, __ATOMIC_RELEASE);
return NULL;
}
case OP_SIGFD: { /* shard 0 only */
struct signalfd_siginfo si;
if (read(sigfd, &si, sizeof si) == sizeof si)
printf("\n[wo-rt-c] signal %u — broadcasting shutdown to %d shards\n",
si.ssi_signo, n_threads);
uint64_t one = 1;
for (int t = 0; t < n_threads; t++)
(void)!write(shards[t].evfd, &one, sizeof one);
break;
}
case OP_ACCEPT: {
if (res >= 0) {
int cfd = res;
if (cfd >= MAX_FDS) close(cfd);
else { conn_open(sh, cfd); arm_recv(sh, cfd); }
}
if (!(cqe->flags & IORING_CQE_F_MORE)) arm_accept(sh); /* re-arm */
break;
}
case OP_RECV: {
struct conn *c = &sh->conns[fd];
if (!c->in_use) break;
if (res <= 0) { conn_close(sh, fd); break; }
c->in_len += (size_t)res;
conn_continue(sh, fd);
break;
}
case OP_SEND: {
struct conn *c = &sh->conns[fd];
if (!c->in_use) break;
if (res <= 0) { conn_close(sh, fd); break; }
c->out_off += (size_t)res;
conn_continue(sh, fd);
break;
}
case OP_WALWR: { /* fd field carries the batch idx */
struct wal_batch *b = &sh->batch[fd];
if (res != (int)b->len)
fprintf(stderr, "[wo-rt-c] shard %d: WAL write %d != %zu\n", sh->id, res, b->len);
else
atomic_fetch_add_explicit(&sh->wal_off, b->len, memory_order_relaxed);
break;
}
case OP_FSYNC: { /* group commit lands: release acks */
struct wal_batch *b = &sh->batch[fd];
int failed = (res < 0); /* incl. -ECANCELED from a failed link */
if (failed)
fprintf(stderr, "[wo-rt-c] shard %d: fsync failed (%d) — dropping %d acks\n",
sh->id, res, b->n_conns);
for (int k = 0; k < b->n_conns; k++) {
int cfd = b->conns[k];
if (cfd < 0 || cfd >= MAX_FDS) continue;
struct conn *c = &sh->conns[cfd];
if (!c->in_use || !c->await_durable) continue;
if (c->gen != b->gens[k]) continue; /* fd reused — not ours */
c->await_durable = 0;
if (failed) conn_close(sh, cfd); /* never ack non-durable */
else conn_continue(sh, cfd);
}
b->len = 0;
b->n_conns = 0;
sh->in_flight = 0;
break;
}
}
}
__atomic_store_n(r->cq_head, head, __ATOMIC_RELEASE);
wal_flush(sh); /* one write→fsync pair per tick */
}
return NULL;
}
/* ------------------------------------------------------------- listener -- */
static int listener_bind(uint16_t port) {
int fd = socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0);
if (fd < 0) { perror("socket"); return -1; }
int one = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);
setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof one);
struct sockaddr_in addr = {0};
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
if (bind(fd, (struct sockaddr *)&addr, sizeof addr) < 0) { perror("bind"); close(fd); return -1; }
if (listen(fd, SOMAXCONN) < 0) { perror("listen"); close(fd); return -1; }
return fd;
}
static int sig_setup(void) {
sigset_t mask;
sigemptyset(&mask);
sigaddset(&mask, SIGINT);
sigaddset(&mask, SIGTERM);
if (sigprocmask(SIG_BLOCK, &mask, NULL) < 0) { perror("sigprocmask"); return -1; }
int fd = signalfd(-1, &mask, SFD_NONBLOCK | SFD_CLOEXEC);
if (fd < 0) perror("signalfd");
return fd;
}
/* ------------------------------------------------------------ wal-check --
* Offline frame walker: validates every record's len/CRC/COMMIT trailer,
* reports the count and where (if anywhere) the log tears. This is the
* crash test's witness, and the skeleton of phase E's replay loop. */
static int wal_check(const char *path) {
int fd = open(path, O_RDONLY);
if (fd < 0) { fprintf(stderr, "wal-check: %s: %s\n", path, strerror(errno)); return 1; }
size_t off = 0;
int recs = 0;
for (;;) {
uint32_t len, crc, end;
struct wal_payload p;
if (pread(fd, &len, 4, (off_t)off) != 4) break;
if (len == 0) break; /* fallocate'd tail */
if (len != sizeof p) {
printf("%s: TORN at byte %zu (bad len %u) — %d whole records before it\n",
path, off, len, recs);
close(fd);
return 0;
}
if (pread(fd, &crc, 4, (off_t)(off + 4)) != 4 ||
pread(fd, &p, sizeof p, (off_t)(off + 8)) != (ssize_t)sizeof p ||
pread(fd, &end, 4, (off_t)(off + 8 + sizeof p)) != 4 ||
crc32(&p, sizeof p) != crc || end != WAL_COMMIT) {
printf("%s: TORN at byte %zu (bad crc/trailer) — %d whole records before it\n",
path, off, recs);
close(fd);
return 0;
}
printf("%s: rec %d op=%u slot=%u id=%d title=\"%s\"\n", path, recs, p.op, p.slot, p.id, p.title);
recs++;
off += WAL_FRAME_BYTES;
}
printf("%s: %d records, all frames valid, clean tail at byte %zu\n", path, recs, off);
close(fd);
return 0;
}
/* ----------------------------------------------------------------- main -- */
int main(int argc, char **argv) {
crc32_init();
if (argc >= 3 && !strcmp(argv[1], "wal-check")) {
int rc = 0;
for (int a = 2; a < argc; a++) rc |= wal_check(argv[a]);
return rc;
}
uint16_t port = 8085;
const char *env = getenv("WO_PORT");
if (env && atoi(env) > 0) port = (uint16_t)atoi(env);
long cores = sysconf(_SC_NPROCESSORS_ONLN);
n_threads = (int)cores;
env = getenv("WO_THREADS");
if (env && atoi(env) > 0) n_threads = atoi(env);
if (n_threads < 1) n_threads = 1;
if (n_threads > MAX_THREADS) n_threads = MAX_THREADS;
/* Million-connection posture: lift the fd ceiling to the hard max. */
struct rlimit rl;
if (getrlimit(RLIMIT_NOFILE, &rl) == 0 && rl.rlim_cur < rl.rlim_max) {
rl.rlim_cur = rl.rlim_max;
setrlimit(RLIMIT_NOFILE, &rl);
}
sigfd = sig_setup();
if (sigfd < 0) return 1;
if (arena_init(n_threads) < 0) return 1;
const char *data_dir = getenv("WO_DATA");
if (!data_dir || !*data_dir) data_dir = "./wo-data";
if (mkdir(data_dir, 0755) < 0 && errno != EEXIST) { perror("mkdir data dir"); return 1; }
/* The data dir is sharded for exactly n_threads. A different WO_THREADS
* would strand WAL/snapshot files silently — refuse (resharding = 09f). */
char mpath[512];
snprintf(mpath, sizeof mpath, "%s/meta", data_dir);
FILE *mf = fopen(mpath, "r");
if (mf) {
int prev = 0;
if (fscanf(mf, "%d", &prev) == 1 && prev != n_threads) {
fprintf(stderr, "[wo-rt-c] %s was written with WO_THREADS=%d — restart with that, or wipe the dir\n",
data_dir, prev);
fclose(mf);
return 1;
}
fclose(mf);
} else if ((mf = fopen(mpath, "w"))) {
fprintf(mf, "%d\n", n_threads);
fclose(mf);
}
shards = calloc((size_t)n_threads, sizeof(struct shard));
if (!shards) { perror("calloc"); return 1; }
for (int t = 0; t < n_threads; t++) {
struct shard *sh = &shards[t];
sh->id = t;
sh->next_id = t + 1;
sh->lfd = listener_bind(port);
sh->evfd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
if (sh->lfd < 0 || sh->evfd < 0) return 1;
if (ring_init(&sh->ring) < 0) return 1;
/* Per-shard WAL, preallocated, NOT truncated — boot replays it. */
char path[512];
snprintf(path, sizeof path, "%s/shard-%d.wal", data_dir, t);
sh->wal_fd = open(path, O_RDWR | O_CREAT | O_CLOEXEC, 0644);
if (sh->wal_fd < 0) { perror("open wal"); return 1; }
if (fallocate(sh->wal_fd, 0, 0, WAL_PREALLOC) < 0)
fprintf(stderr, "[wo-rt-c] warn: fallocate %s refused (%s)\n", path, strerror(errno));
snprintf(sh->snap_path, sizeof sh->snap_path, "%s/shard-%d.data", data_dir, t);
}
printf("[wo-rt-c] %d shard%s on http://127.0.0.1:%u — io_uring loops, keep-alive — arena %zu KB (%s pages, %s) — ctrl-C to stop\n",
n_threads, n_threads == 1 ? "" : "s", port,
arena_map_bytes / 1024,
arena_huge ? "2M huge" : "4K",
arena_locked ? "mlocked" : "NOT locked");
printf(" GET / runtime + arena info, per-shard stats\n GET /healthz liveness\n");
printf(" GET /api/notes list (connection's shard)\n POST /api/notes create {\"title\":\"...\"}\n");
fflush(stdout);
for (int t = 0; t < n_threads; t++)
pthread_create(&shards[t].tid, NULL, shard_main, &shards[t]);
for (int t = 0; t < n_threads; t++)
pthread_join(shards[t].tid, NULL);
/* Clean shutdown: persist each slice as a snapshot, truncate the WALs.
* A kill -9 skips this — that's what boot-time WAL replay is for. */
for (int t = 0; t < n_threads; t++)
snap_write(&shards[t]);
printf("[wo-rt-c] all %d shards joined — bye\n", n_threads);
for (int t = 0; t < n_threads; t++) {
close(shards[t].ring.fd); close(shards[t].lfd); close(shards[t].evfd);
close(shards[t].wal_fd);
}
close(sigfd);
munmap(arena, arena_map_bytes);
free(shards);
return 0;
}

51
reference/README.md
View file

@ -1,51 +0,0 @@
# `reference/` — archived for reference
Source material preserved alongside the active codebase. Nothing here is on the build path of the root workspace.
## `reference/crates/` — v1 writeonce blog
The original writeonce blog engine: 13 Rust crates implementing a flat-file content store, three sidecar indexes, a minimal HTTP server, and an `.htmlx` template layer.
| Crate | Responsibility |
| --- | --- |
| `wo-model` | `Article` domain type + content loader |
| `wo-seg` | Append-only `.seg` binary file format (~475 LOC) |
| `wo-index` | `title.idx`, `date.idx`, `tags.idx` sidecar indexes |
| `wo-store` | Composes seg + indexes, exposes the query API |
| `wo-watch` | inotify-driven content ingest |
| `wo-event` | Domain event type |
| `wo-sub` | Subscription layer (pre-engine-native `LIVE`) |
| `wo-rt` | v1 runtime glue |
| `wo-http`, `wo-route`, `wo-serve` | HTTP server |
| `wo-htmlx` | `.htmlx` template engine |
| `wo-md` | Markdown rendering |
It is a nested workspace. Build or test it standalone:
```bash
cd reference/crates
cargo build
cargo test
```
See [docs/runtime/database/07-wo-seg-migration.md](../docs/runtime/database/07-wo-seg-migration.md) for the plan to replace these crates with the new `.wo` runtime at `crates/wo-rt/` via phased coexistence. The v1 codebase is the source that migration reads from.
## `reference/writeonce-api/` and `reference/writeonce-app/`
Earlier exploration snapshots that predate the v1 crates. Self-contained; no active build wiring.
## Research source trees (symlinks)
Gitignored, user-specific absolute paths — each contributor sets their own:
```bash
ln -s <path-to-linux-src> reference/linux
ln -s <path-to-go-src> reference/go
ln -s <path-to-postgresql-src> reference/postgresql
ln -s <path-to-mcp-python-sdk-src> reference/mcp-python-sdk # https://github.com/modelcontextprotocol/python-sdk.git
```
- **`reference/linux/`** — the Linux kernel source. Read `io_uring/`, `fs/notify/inotify/`, `kernel/eventfd.c`, `include/uapi/linux/*.h` when designing kernel-primitive modules. See per-primitive reference cards under [`docs/plan/linux/`](../docs/plan/linux/).
- **`reference/go/`** — the Go source tree. Read `src/runtime/netpoll_epoll.go`, `src/runtime/netpoll.go`, `src/runtime/asm_*.s`, `src/runtime/sys_linux_*.s` when designing the runtime layer — writeonce's `crates/rt/src/runtime/` mirrors Go's `src/runtime/` file-per-flavour naming. See [`docs/plan/assembly/`](../docs/plan/assembly/) for the asm policy doc that cites this tree.
- **`reference/postgresql/`** — the PostgreSQL source tree, for database-engine research.
- **`reference/mcp-python-sdk/`** — the official MCP Python SDK, the reference implementation for [plan 15](../docs/plan/15-mcp-streamable-http.md) (MCP over Streamable HTTP). Read `src/mcp/server/streamable_http.py` (transport server: POST/GET handling, SSE framing, session validation, `Last-Event-ID` replay) and `src/mcp/server/streamable_http_manager.py` (session lifecycle) when implementing the server side; `src/mcp/client/streamable_http.py` shows what a conforming client expects. writeonce ports the *behaviour*, not the code — the Rust implementation stays hand-rolled per the zero-deps doctrine.

3
reference/crates/.gitignore vendored
View file

@ -1,3 +0,0 @@
# Nested v1 workspace — has its own Cargo.lock and target dir.
/target
Cargo.lock

40
reference/crates/Cargo.toml
View file

@ -1,40 +0,0 @@
# Nested workspace for the v1 writeonce blog crates.
#
# The root workspace at /Cargo.toml drives the new `.wo` runtime at crates/wo-rt.
# These thirteen crates are the previous generation of writeonce — the flat-file
# blog engine described in docs/runtime/database/07-wo-seg-migration.md. They
# are preserved here for reference: source is readable, tests still run, and
# the Phase 7 migration plan assumes a gradual trait-abstracted coexistence
# rather than deletion.
#
# To build the v1 system standalone:
# cd reference/crates && cargo build
# cd reference/crates && cargo test
#
# The v1 crates are deliberately NOT members of the root workspace. Excluding
# them avoids the `wo-rt` name collision (both v1 and v2 claim that name) and
# keeps the root build fast while the new runtime is under active development.
[workspace]
resolver = "2"
members = [
"wo-model",
"wo-seg",
"wo-index",
"wo-store",
"wo-watch",
"wo-event",
"wo-sub",
"wo-rt",
"wo-http",
"wo-route",
"wo-htmlx",
"wo-md",
"wo-serve",
]
[workspace.dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
bincode = "1"
libc = "0.2"

7
reference/crates/wo-event/Cargo.toml
View file

@ -1,7 +0,0 @@
[package]
name = "wo-event"
version = "0.1.0"
edition = "2021"
[dependencies]
libc = { workspace = true }

183
reference/crates/wo-event/src/epoll.rs
View file

@ -1,183 +0,0 @@
use std::io;
use std::os::unix::io::RawFd;
use std::time::Duration;
/// Caller-assigned identifier for a registered file descriptor.
pub type Token = u64;
/// Interest flags for epoll registration.
#[derive(Debug, Clone, Copy)]
pub enum Interest {
Readable,
Writable,
ReadWrite,
}
impl Interest {
fn to_epoll_flags(self) -> u32 {
match self {
Interest::Readable => libc::EPOLLIN as u32,
Interest::Writable => libc::EPOLLOUT as u32,
Interest::ReadWrite => (libc::EPOLLIN | libc::EPOLLOUT) as u32,
}
}
}
/// An event delivered by the event loop.
#[derive(Debug, Clone)]
pub struct Event {
pub token: Token,
pub readable: bool,
pub writable: bool,
pub error: bool,
pub hangup: bool,
}
/// Single-threaded event loop built on epoll.
pub struct EventLoop {
epoll_fd: RawFd,
}
impl EventLoop {
/// Create a new event loop.
pub fn new() -> io::Result<Self> {
let fd = unsafe { libc::epoll_create1(libc::EPOLL_CLOEXEC) };
if fd < 0 {
return Err(io::Error::last_os_error());
}
Ok(Self { epoll_fd: fd })
}
/// Register a file descriptor for the given interest.
pub fn register(&self, fd: RawFd, interest: Interest, token: Token) -> io::Result<()> {
let mut event = libc::epoll_event {
events: interest.to_epoll_flags() | libc::EPOLLRDHUP as u32,
u64: token,
};
let ret = unsafe { libc::epoll_ctl(self.epoll_fd, libc::EPOLL_CTL_ADD, fd, &mut event) };
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(())
}
}
/// Modify interest for an already-registered fd.
pub fn modify(&self, fd: RawFd, interest: Interest, token: Token) -> io::Result<()> {
let mut event = libc::epoll_event {
events: interest.to_epoll_flags() | libc::EPOLLRDHUP as u32,
u64: token,
};
let ret = unsafe { libc::epoll_ctl(self.epoll_fd, libc::EPOLL_CTL_MOD, fd, &mut event) };
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(())
}
}
/// Remove a file descriptor from the event loop.
pub fn deregister(&self, fd: RawFd) -> io::Result<()> {
let ret = unsafe {
libc::epoll_ctl(self.epoll_fd, libc::EPOLL_CTL_DEL, fd, std::ptr::null_mut())
};
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(())
}
}
/// Wait for events. Returns when at least one event is ready or timeout expires.
///
/// `timeout`: `None` blocks indefinitely, `Some(duration)` sets a timeout.
/// Returns up to 64 events per call.
pub fn poll(&self, timeout: Option<Duration>) -> io::Result<Vec<Event>> {
let timeout_ms = match timeout {
None => -1i32,
Some(d) => d.as_millis() as i32,
};
let mut events = [libc::epoll_event { events: 0, u64: 0 }; 64];
let n = unsafe {
libc::epoll_wait(self.epoll_fd, events.as_mut_ptr(), events.len() as i32, timeout_ms)
};
if n < 0 {
let err = io::Error::last_os_error();
// EINTR is not an error — just return empty.
if err.raw_os_error() == Some(libc::EINTR) {
return Ok(vec![]);
}
return Err(err);
}
let result = (0..n as usize)
.map(|i| {
let e = events[i].events;
Event {
token: events[i].u64,
readable: (e & libc::EPOLLIN as u32) != 0,
writable: (e & libc::EPOLLOUT as u32) != 0,
error: (e & libc::EPOLLERR as u32) != 0,
hangup: (e & (libc::EPOLLHUP | libc::EPOLLRDHUP) as u32) != 0,
}
})
.collect();
Ok(result)
}
/// Get the raw epoll file descriptor.
pub fn fd(&self) -> RawFd {
self.epoll_fd
}
}
impl Drop for EventLoop {
fn drop(&mut self) {
unsafe { libc::close(self.epoll_fd) };
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::EventFd;
#[test]
fn register_and_poll_eventfd() {
let eloop = EventLoop::new().unwrap();
let efd = EventFd::new().unwrap();
eloop.register(efd.fd(), Interest::Readable, 42).unwrap();
// Write to eventfd from the same thread.
efd.write(1).unwrap();
let events = eloop.poll(Some(Duration::from_millis(100))).unwrap();
assert_eq!(events.len(), 1);
assert_eq!(events[0].token, 42);
assert!(events[0].readable);
}
#[test]
fn poll_timeout_no_events() {
let eloop = EventLoop::new().unwrap();
let events = eloop.poll(Some(Duration::from_millis(10))).unwrap();
assert!(events.is_empty());
}
#[test]
fn deregister() {
let eloop = EventLoop::new().unwrap();
let efd = EventFd::new().unwrap();
eloop.register(efd.fd(), Interest::Readable, 1).unwrap();
eloop.deregister(efd.fd()).unwrap();
efd.write(1).unwrap();
let events = eloop.poll(Some(Duration::from_millis(10))).unwrap();
assert!(events.is_empty());
}
}

66
reference/crates/wo-event/src/eventfd.rs
View file

@ -1,66 +0,0 @@
use std::io;
use std::os::unix::io::RawFd;
/// Wrapper around Linux eventfd(2) for lightweight signaling.
pub struct EventFd {
fd: RawFd,
}
impl EventFd {
/// Create a new eventfd with initial value 0.
pub fn new() -> io::Result<Self> {
let fd = unsafe { libc::eventfd(0, libc::EFD_NONBLOCK | libc::EFD_CLOEXEC) };
if fd < 0 {
return Err(io::Error::last_os_error());
}
Ok(Self { fd })
}
/// Write a value to the eventfd (signals waiters).
pub fn write(&self, val: u64) -> io::Result<()> {
let buf = val.to_ne_bytes();
let ret = unsafe { libc::write(self.fd, buf.as_ptr() as *const libc::c_void, 8) };
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(())
}
}
/// Read the current value (resets to 0).
pub fn read(&self) -> io::Result<u64> {
let mut buf = [0u8; 8];
let ret = unsafe { libc::read(self.fd, buf.as_mut_ptr() as *mut libc::c_void, 8) };
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(u64::from_ne_bytes(buf))
}
}
/// Get the raw file descriptor for epoll registration.
pub fn fd(&self) -> RawFd {
self.fd
}
}
impl Drop for EventFd {
fn drop(&mut self) {
unsafe { libc::close(self.fd) };
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn write_and_read() {
let efd = EventFd::new().unwrap();
efd.write(5).unwrap();
efd.write(3).unwrap();
// eventfd accumulates writes.
let val = efd.read().unwrap();
assert_eq!(val, 8);
}
}

9
reference/crates/wo-event/src/lib.rs
View file

@ -1,9 +0,0 @@
mod epoll;
mod eventfd;
mod timerfd;
mod signalfd;
pub use epoll::{EventLoop, Event, Interest, Token};
pub use eventfd::EventFd;
pub use timerfd::TimerFd;
pub use signalfd::SignalFd;

62
reference/crates/wo-event/src/signalfd.rs
View file

@ -1,62 +0,0 @@
use std::io;
use std::os::unix::io::RawFd;
/// Wrapper around Linux signalfd(2) for handling signals via fd.
///
/// Blocks SIGINT and SIGTERM in the process signal mask and delivers
/// them as readable events on the fd instead.
pub struct SignalFd {
fd: RawFd,
}
impl SignalFd {
/// Create a signalfd that catches SIGINT and SIGTERM.
///
/// Also blocks these signals in the process signal mask so they
/// don't trigger default handlers.
pub fn new() -> io::Result<Self> {
let mut mask: libc::sigset_t = unsafe { std::mem::zeroed() };
unsafe {
libc::sigemptyset(&mut mask);
libc::sigaddset(&mut mask, libc::SIGINT);
libc::sigaddset(&mut mask, libc::SIGTERM);
// Block these signals so they go to signalfd instead.
let ret = libc::pthread_sigmask(libc::SIG_BLOCK, &mask, std::ptr::null_mut());
if ret != 0 {
return Err(io::Error::from_raw_os_error(ret));
}
}
let fd = unsafe { libc::signalfd(-1, &mask, libc::SFD_NONBLOCK | libc::SFD_CLOEXEC) };
if fd < 0 {
return Err(io::Error::last_os_error());
}
Ok(Self { fd })
}
/// Read a pending signal. Returns the signal number (e.g., SIGINT = 2).
pub fn read(&self) -> io::Result<i32> {
let mut info: libc::signalfd_siginfo = unsafe { std::mem::zeroed() };
let size = std::mem::size_of::<libc::signalfd_siginfo>();
let ret = unsafe {
libc::read(self.fd, &mut info as *mut _ as *mut libc::c_void, size)
};
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(info.ssi_signo as i32)
}
}
pub fn fd(&self) -> RawFd {
self.fd
}
}
impl Drop for SignalFd {
fn drop(&mut self) {
unsafe { libc::close(self.fd) };
}
}

91
reference/crates/wo-event/src/timerfd.rs
View file

@ -1,91 +0,0 @@
use std::io;
use std::os::unix::io::RawFd;
use std::time::Duration;
/// Wrapper around Linux timerfd_create(2) for timer-as-fd.
pub struct TimerFd {
fd: RawFd,
}
impl TimerFd {
/// Create a new monotonic timer fd.
pub fn new() -> io::Result<Self> {
let fd = unsafe {
libc::timerfd_create(
libc::CLOCK_MONOTONIC,
libc::TFD_NONBLOCK | libc::TFD_CLOEXEC,
)
};
if fd < 0 {
return Err(io::Error::last_os_error());
}
Ok(Self { fd })
}
/// Arm the timer to fire once after `initial` and then repeat every `interval`.
///
/// Pass `Duration::ZERO` for `interval` for a one-shot timer.
pub fn set(&self, initial: Duration, interval: Duration) -> io::Result<()> {
let spec = libc::itimerspec {
it_interval: duration_to_timespec(interval),
it_value: duration_to_timespec(initial),
};
let ret =
unsafe { libc::timerfd_settime(self.fd, 0, &spec, std::ptr::null_mut()) };
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(())
}
}
/// Read the number of expirations since last read.
pub fn read(&self) -> io::Result<u64> {
let mut buf = [0u8; 8];
let ret = unsafe { libc::read(self.fd, buf.as_mut_ptr() as *mut libc::c_void, 8) };
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(u64::from_ne_bytes(buf))
}
}
pub fn fd(&self) -> RawFd {
self.fd
}
}
impl Drop for TimerFd {
fn drop(&mut self) {
unsafe { libc::close(self.fd) };
}
}
fn duration_to_timespec(d: Duration) -> libc::timespec {
libc::timespec {
tv_sec: d.as_secs() as libc::time_t,
tv_nsec: d.subsec_nanos() as libc::c_long,
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{EventLoop, Interest};
#[test]
fn timer_fires() {
let eloop = EventLoop::new().unwrap();
let timer = TimerFd::new().unwrap();
timer.set(Duration::from_millis(20), Duration::ZERO).unwrap();
eloop.register(timer.fd(), Interest::Readable, 99).unwrap();
let events = eloop.poll(Some(Duration::from_millis(100))).unwrap();
assert!(!events.is_empty());
assert_eq!(events[0].token, 99);
let expirations = timer.read().unwrap();
assert!(expirations >= 1);
}
}

13
reference/crates/wo-htmlx/Cargo.toml
View file

@ -1,13 +0,0 @@
[package]
name = "wo-htmlx"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-model = { path = "../wo-model" }
wo-md = { path = "../wo-md" }
serde = { workspace = true }
serde_json = { workspace = true }
[dev-dependencies]
tempfile = "3"

18
reference/crates/wo-htmlx/src/ast.rs
View file

@ -1,18 +0,0 @@
/// A parsed template node.
#[derive(Debug, Clone, PartialEq)]
pub enum Node {
/// Raw HTML passthrough.
Literal(String),
/// Variable binding: `{{path.to.value}}`
Binding(Vec<String>),
/// Iteration: `{{#each path}} ... {{/each}}`
Each {
path: Vec<String>,
body: Vec<Node>,
},
/// Partial inclusion: `{{> name arg=value}}`
Partial {
name: String,
args: Vec<(String, Vec<String>)>,
},
}

11
reference/crates/wo-htmlx/src/lib.rs
View file

@ -1,11 +0,0 @@
mod ast;
mod parser;
mod render;
mod registry;
mod value;
pub use ast::Node;
pub use parser::parse;
pub use render::render;
pub use registry::TemplateRegistry;
pub use value::Value;

173
reference/crates/wo-htmlx/src/parser.rs
View file

@ -1,173 +0,0 @@
use crate::ast::Node;
/// Parse a template string into a list of AST nodes.
pub fn parse(template: &str) -> Vec<Node> {
let mut nodes = Vec::new();
let mut rest = template;
while !rest.is_empty() {
if let Some(pos) = rest.find("{{") {
// Literal before the opening `{{`.
if pos > 0 {
nodes.push(Node::Literal(rest[..pos].to_string()));
}
let after_open = &rest[pos + 2..];
if let Some(close) = after_open.find("}}") {
let expr = after_open[..close].trim();
rest = &after_open[close + 2..];
if let Some(each_path) = expr.strip_prefix("#each ") {
// Block: {{#each path}} ... {{/each}}
let path = parse_path(each_path.trim());
let (body, remaining) = parse_until_end_each(rest);
nodes.push(Node::Each { path, body });
rest = remaining;
} else if let Some(partial_expr) = expr.strip_prefix("> ") {
// Partial: {{> name arg=value}}
let (name, args) = parse_partial_expr(partial_expr.trim());
nodes.push(Node::Partial { name, args });
} else if !expr.starts_with('/') {
// Binding: {{path.to.value}}
let path = parse_path(expr);
nodes.push(Node::Binding(path));
}
// {{/each}} handled by parse_until_end_each
} else {
// No closing `}}` — treat rest as literal.
nodes.push(Node::Literal(rest.to_string()));
break;
}
} else {
// No more `{{` — rest is literal.
nodes.push(Node::Literal(rest.to_string()));
break;
}
}
nodes
}
fn parse_path(s: &str) -> Vec<String> {
s.split('.').map(|p| p.trim().to_string()).collect()
}
fn parse_until_end_each(input: &str) -> (Vec<Node>, &str) {
// Find the matching {{/each}}.
let mut depth = 1;
let mut pos = 0;
let bytes = input.as_bytes();
while pos < bytes.len() {
if let Some(open) = input[pos..].find("{{") {
let abs = pos + open;
let after = &input[abs + 2..];
if let Some(close) = after.find("}}") {
let expr = after[..close].trim();
if expr.starts_with("#each ") {
depth += 1;
} else if expr == "/each" {
depth -= 1;
if depth == 0 {
let body_str = &input[..abs];
let rest = &after[close + 2..];
return (parse(body_str), rest);
}
}
pos = abs + 2 + close + 2;
} else {
break;
}
} else {
break;
}
}
// Unmatched {{#each}} — return rest as literal body.
(vec![Node::Literal(input.to_string())], "")
}
fn parse_partial_expr(expr: &str) -> (String, Vec<(String, Vec<String>)>) {
let parts: Vec<&str> = expr.splitn(2, ' ').collect();
let name = parts[0].to_string();
let mut args = Vec::new();
if parts.len() > 1 {
for arg in parts[1].split_whitespace() {
if let Some((key, val)) = arg.split_once('=') {
args.push((key.to_string(), parse_path(val)));
}
}
}
(name, args)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ast::Node;
#[test]
fn parse_literal() {
let nodes = parse("<h1>Hello</h1>");
assert_eq!(nodes, vec![Node::Literal("<h1>Hello</h1>".into())]);
}
#[test]
fn parse_binding() {
let nodes = parse("{{article.title}}");
assert_eq!(nodes, vec![Node::Binding(vec!["article".into(), "title".into()])]);
}
#[test]
fn parse_mixed() {
let nodes = parse("<h1>{{title}}</h1>");
assert_eq!(nodes.len(), 3);
assert_eq!(nodes[0], Node::Literal("<h1>".into()));
assert_eq!(nodes[1], Node::Binding(vec!["title".into()]));
assert_eq!(nodes[2], Node::Literal("</h1>".into()));
}
#[test]
fn parse_each() {
let nodes = parse("{{#each items}}<li>{{this}}</li>{{/each}}");
assert_eq!(nodes.len(), 1);
match &nodes[0] {
Node::Each { path, body } => {
assert_eq!(path, &vec!["items".to_string()]);
assert_eq!(body.len(), 3);
}
_ => panic!("expected Each"),
}
}
#[test]
fn parse_partial() {
let nodes = parse("{{> article-card article=this}}");
assert_eq!(nodes.len(), 1);
match &nodes[0] {
Node::Partial { name, args } => {
assert_eq!(name, "article-card");
assert_eq!(args.len(), 1);
assert_eq!(args[0].0, "article");
assert_eq!(args[0].1, vec!["this".to_string()]);
}
_ => panic!("expected Partial"),
}
}
#[test]
fn parse_nested_each() {
let nodes = parse("{{#each sections}}{{#each paragraphs}}{{this}}{{/each}}{{/each}}");
assert_eq!(nodes.len(), 1);
match &nodes[0] {
Node::Each { body, .. } => {
assert_eq!(body.len(), 1);
assert!(matches!(&body[0], Node::Each { .. }));
}
_ => panic!("expected nested Each"),
}
}
}

121
reference/crates/wo-htmlx/src/registry.rs
View file

@ -1,121 +0,0 @@
use std::collections::HashMap;
use std::fs;
use std::io;
use std::path::Path;
use crate::ast::Node;
use crate::parser;
/// Holds parsed templates and partials, loaded from a templates directory.
pub struct TemplateRegistry {
/// Page templates (e.g., "home" -> parsed nodes from home.htmlx).
templates: HashMap<String, Vec<Node>>,
/// Component partials (e.g., "article-card" -> parsed nodes).
partials: HashMap<String, Vec<Node>>,
}
impl TemplateRegistry {
/// Load all .htmlx files from the given templates directory.
///
/// Templates in the root are page templates.
/// Templates in `components/` are partials.
pub fn load(templates_dir: &Path) -> io::Result<Self> {
let mut templates = HashMap::new();
let mut partials = HashMap::new();
// Load page templates from root.
if templates_dir.exists() {
for entry in fs::read_dir(templates_dir)? {
let entry = entry?;
let path = entry.path();
if path.is_file() && has_htmlx_ext(&path) {
let name = stem(&path);
let content = fs::read_to_string(&path)?;
templates.insert(name, parser::parse(&content));
}
}
}
// Load partials from components/.
let components_dir = templates_dir.join("components");
if components_dir.exists() {
for entry in fs::read_dir(&components_dir)? {
let entry = entry?;
let path = entry.path();
if path.is_file() && has_htmlx_ext(&path) {
let name = stem(&path);
let content = fs::read_to_string(&path)?;
partials.insert(name, parser::parse(&content));
}
}
}
Ok(Self { templates, partials })
}
/// Get a page template by name.
pub fn get(&self, name: &str) -> Option<&Vec<Node>> {
self.templates.get(name)
}
/// Get the partials map (for passing to render()).
pub fn partials(&self) -> &HashMap<String, Vec<Node>> {
&self.partials
}
/// Number of loaded templates.
pub fn template_count(&self) -> usize {
self.templates.len()
}
/// Number of loaded partials.
pub fn partial_count(&self) -> usize {
self.partials.len()
}
}
fn has_htmlx_ext(path: &Path) -> bool {
path.extension().and_then(|e| e.to_str()) == Some("htmlx")
}
fn stem(path: &Path) -> String {
path.file_stem()
.and_then(|s| s.to_str())
.unwrap_or("")
.to_string()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn load_templates() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path();
fs::write(dir.join("home.htmlx"), "<h1>{{title}}</h1>").unwrap();
fs::write(dir.join("article.htmlx"), "<article>{{body}}</article>").unwrap();
fs::create_dir_all(dir.join("components")).unwrap();
fs::write(
dir.join("components/card.htmlx"),
"<div>{{name}}</div>",
)
.unwrap();
let reg = TemplateRegistry::load(dir).unwrap();
assert_eq!(reg.template_count(), 2);
assert_eq!(reg.partial_count(), 1);
assert!(reg.get("home").is_some());
assert!(reg.get("article").is_some());
assert!(reg.partials().contains_key("card"));
}
#[test]
fn empty_directory() {
let tmp = tempfile::tempdir().unwrap();
let reg = TemplateRegistry::load(tmp.path()).unwrap();
assert_eq!(reg.template_count(), 0);
}
}

140
reference/crates/wo-htmlx/src/render.rs
View file

@ -1,140 +0,0 @@
use std::collections::BTreeMap;
use crate::ast::Node;
use crate::value::Value;
/// Render a list of AST nodes against a context value.
///
/// `partials` maps partial names to their parsed node lists.
pub fn render(
nodes: &[Node],
context: &Value,
partials: &std::collections::HashMap<String, Vec<Node>>,
) -> String {
let mut output = String::new();
for node in nodes {
match node {
Node::Literal(text) => {
output.push_str(text);
}
Node::Binding(path) => {
if path.len() == 1 && path[0] == "this" {
output.push_str(&context.to_display());
} else {
let val = context.resolve(path);
output.push_str(&val.to_display());
}
}
Node::Each { path, body } => {
let list_val = if path.len() == 1 && path[0] == "this" {
context
} else {
context.resolve(path)
};
for item in list_val.as_list() {
output.push_str(&render(body, item, partials));
}
}
Node::Partial { name, args } => {
if let Some(partial_nodes) = partials.get(name.as_str()) {
// Build the partial context from args.
let partial_ctx = if args.is_empty() {
context.clone()
} else {
let mut map = BTreeMap::new();
for (key, path) in args {
let val = if path.len() == 1 && path[0] == "this" {
context.clone()
} else {
context.resolve(path).clone()
};
map.insert(key.clone(), val);
}
Value::Object(map)
};
output.push_str(&render(partial_nodes, &partial_ctx, partials));
}
}
}
}
output
}
#[cfg(test)]
mod tests {
use super::*;
use crate::parser;
use std::collections::HashMap;
fn ctx(json: serde_json::Value) -> Value {
Value::from_json(&json)
}
#[test]
fn render_literal() {
let nodes = parser::parse("<h1>Hello</h1>");
let result = render(&nodes, &Value::Null, &HashMap::new());
assert_eq!(result, "<h1>Hello</h1>");
}
#[test]
fn render_binding() {
let nodes = parser::parse("<h1>{{title}}</h1>");
let context = ctx(serde_json::json!({"title": "My Article"}));
let result = render(&nodes, &context, &HashMap::new());
assert_eq!(result, "<h1>My Article</h1>");
}
#[test]
fn render_nested_binding() {
let nodes = parser::parse("by {{article.author}}");
let context = ctx(serde_json::json!({"article": {"author": "Shoney"}}));
let result = render(&nodes, &context, &HashMap::new());
assert_eq!(result, "by Shoney");
}
#[test]
fn render_each() {
let nodes = parser::parse("<ul>{{#each tags}}<li>{{this}}</li>{{/each}}</ul>");
let context = ctx(serde_json::json!({"tags": ["rust", "linux"]}));
let result = render(&nodes, &context, &HashMap::new());
assert_eq!(result, "<ul><li>rust</li><li>linux</li></ul>");
}
#[test]
fn render_each_objects() {
let nodes = parser::parse("{{#each sections}}<h2>{{heading}}</h2>{{/each}}");
let context = ctx(serde_json::json!({
"sections": [
{"heading": "Intro"},
{"heading": "Details"}
]
}));
let result = render(&nodes, &context, &HashMap::new());
assert_eq!(result, "<h2>Intro</h2><h2>Details</h2>");
}
#[test]
fn render_partial() {
let nodes = parser::parse("{{> card item=this}}");
let mut partials = HashMap::new();
partials.insert(
"card".to_string(),
parser::parse("<div>{{item.name}}</div>"),
);
let context = ctx(serde_json::json!({"name": "Test"}));
let result = render(&nodes, &context, &partials);
assert_eq!(result, "<div>Test</div>");
}
#[test]
fn render_missing_binding() {
let nodes = parser::parse("{{nonexistent}}");
let context = ctx(serde_json::json!({}));
let result = render(&nodes, &context, &HashMap::new());
assert_eq!(result, "");
}
}

122
reference/crates/wo-htmlx/src/value.rs
View file

@ -1,122 +0,0 @@
use std::collections::BTreeMap;
/// Template context value — the data that template bindings resolve against.
#[derive(Debug, Clone, PartialEq)]
pub enum Value {
String(String),
Number(f64),
Bool(bool),
List(Vec<Value>),
Object(BTreeMap<String, Value>),
Null,
}
impl Value {
/// Resolve a dotted path like ["article", "title"] against this value.
pub fn resolve(&self, path: &[String]) -> &Value {
let mut current = self;
for key in path {
match current {
Value::Object(map) => {
current = map.get(key.as_str()).unwrap_or(&Value::Null);
}
_ => return &Value::Null,
}
}
current
}
/// Convert to a display string for template output.
pub fn to_display(&self) -> String {
match self {
Value::String(s) => s.clone(),
Value::Number(n) => {
if *n == (*n as i64) as f64 {
format!("{}", *n as i64)
} else {
format!("{}", n)
}
}
Value::Bool(b) => b.to_string(),
Value::Null => String::new(),
Value::List(items) => {
let parts: Vec<String> = items.iter().map(|v| v.to_display()).collect();
parts.join(", ")
}
Value::Object(_) => "[object]".to_string(),
}
}
/// Convert to a list for iteration (returns empty vec if not a list).
pub fn as_list(&self) -> &[Value] {
match self {
Value::List(items) => items,
_ => &[],
}
}
/// Convert a serde_json::Value to a template Value.
pub fn from_json(json: &serde_json::Value) -> Self {
match json {
serde_json::Value::Null => Value::Null,
serde_json::Value::Bool(b) => Value::Bool(*b),
serde_json::Value::Number(n) => Value::Number(n.as_f64().unwrap_or(0.0)),
serde_json::Value::String(s) => Value::String(s.clone()),
serde_json::Value::Array(arr) => {
Value::List(arr.iter().map(Value::from_json).collect())
}
serde_json::Value::Object(map) => {
let btree = map
.iter()
.map(|(k, v)| (k.clone(), Value::from_json(v)))
.collect();
Value::Object(btree)
}
}
}
}
/// Convert an Article to a template Value using serde_json as intermediary.
pub fn article_to_value(article: &wo_model::Article) -> Value {
let json = serde_json::to_value(article).unwrap_or(serde_json::Value::Null);
Value::from_json(&json)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resolve_path() {
let mut inner = BTreeMap::new();
inner.insert("title".into(), Value::String("Hello".into()));
let mut root = BTreeMap::new();
root.insert("article".into(), Value::Object(inner));
let val = Value::Object(root);
let result = val.resolve(&["article".into(), "title".into()]);
assert_eq!(result, &Value::String("Hello".into()));
}
#[test]
fn resolve_missing() {
let val = Value::Object(BTreeMap::new());
assert_eq!(val.resolve(&["nope".into()]), &Value::Null);
}
#[test]
fn from_json() {
let json: serde_json::Value = serde_json::json!({
"name": "test",
"count": 42,
"active": true,
"tags": ["a", "b"]
});
let val = Value::from_json(&json);
assert_eq!(
val.resolve(&["name".into()]),
&Value::String("test".into())
);
assert_eq!(val.resolve(&["count".into()]), &Value::Number(42.0));
}
}

10
reference/crates/wo-http/Cargo.toml
View file

@ -1,10 +0,0 @@
[package]
name = "wo-http"
version = "0.1.0"
edition = "2021"
[dependencies]
libc = { workspace = true }
[dev-dependencies]
tempfile = "3"

202
reference/crates/wo-http/src/connection.rs
View file

@ -1,202 +0,0 @@
use std::io;
use std::os::unix::io::RawFd;
use crate::request::{self, ParseResult, Request};
use crate::response::Response;
/// Connection state machine.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum State {
/// Accumulating request bytes.
ReadingRequest,
/// Writing response bytes.
WritingResponse,
/// Initial response sent, fd handed to subscription manager.
/// Connection stays open for diff payloads.
Subscribed,
/// Ready to close.
Done,
}
/// A single HTTP connection.
pub struct Connection {
pub fd: RawFd,
pub state: State,
read_buf: Vec<u8>,
write_buf: Vec<u8>,
write_offset: usize,
parsed_request: Option<Request>,
}
impl Connection {
pub fn new(fd: RawFd) -> Self {
Self {
fd,
state: State::ReadingRequest,
read_buf: Vec::with_capacity(4096),
write_buf: Vec::new(),
write_offset: 0,
parsed_request: None,
}
}
/// Read available data from the socket into the read buffer.
/// Returns Ok(true) if data was read, Ok(false) if connection closed.
pub fn read(&mut self) -> io::Result<bool> {
let mut buf = [0u8; 4096];
let n = unsafe {
libc::read(self.fd, buf.as_mut_ptr() as *mut libc::c_void, buf.len())
};
if n < 0 {
let err = io::Error::last_os_error();
if err.raw_os_error() == Some(libc::EAGAIN) {
return Ok(true); // no data yet, but connection still open
}
return Err(err);
}
if n == 0 {
return Ok(false); // connection closed
}
self.read_buf.extend_from_slice(&buf[..n as usize]);
Ok(true)
}
/// Try to parse the request from the read buffer.
/// Returns Some(Request) if a full request was parsed.
pub fn try_parse(&mut self) -> Option<Request> {
if self.parsed_request.is_some() {
return self.parsed_request.clone();
}
match request::parse(&self.read_buf) {
ParseResult::Complete(req, _consumed) => {
self.parsed_request = Some(req.clone());
Some(req)
}
ParseResult::Incomplete => None,
ParseResult::Error(_) => {
self.state = State::Done;
None
}
}
}
/// Set the response to write back to the client.
pub fn set_response(&mut self, response: &Response) {
self.write_buf = response.to_bytes();
self.write_offset = 0;
self.state = State::WritingResponse;
}
/// Write pending response bytes to the socket.
/// Returns Ok(true) if all bytes written, Ok(false) if partial (need EPOLLOUT).
pub fn write(&mut self) -> io::Result<bool> {
let remaining = &self.write_buf[self.write_offset..];
if remaining.is_empty() {
return Ok(true);
}
let n = unsafe {
libc::write(
self.fd,
remaining.as_ptr() as *const libc::c_void,
remaining.len(),
)
};
if n < 0 {
let err = io::Error::last_os_error();
if err.raw_os_error() == Some(libc::EAGAIN) {
return Ok(false);
}
return Err(err);
}
self.write_offset += n as usize;
if self.write_offset >= self.write_buf.len() {
Ok(true) // all written
} else {
Ok(false) // partial write
}
}
/// Transition to subscribed state (connection stays open).
pub fn set_subscribed(&mut self) {
self.state = State::Subscribed;
}
/// Mark connection as done (will be closed).
pub fn set_done(&mut self) {
self.state = State::Done;
}
/// Take the parsed request out of the connection.
pub fn take_request(&mut self) -> Option<Request> {
self.parsed_request.take()
}
}
impl Drop for Connection {
fn drop(&mut self) {
unsafe { libc::close(self.fd) };
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn connection_lifecycle() {
// Use a socketpair to simulate a connection.
let mut fds = [0i32; 2];
assert_eq!(
unsafe { libc::socketpair(libc::AF_UNIX, libc::SOCK_STREAM | libc::SOCK_NONBLOCK, 0, fds.as_mut_ptr()) },
0
);
let (server_fd, client_fd) = (fds[0], fds[1]);
// Write a request from the "client" side.
let request_bytes = b"GET /blog/test HTTP/1.1\r\nHost: localhost\r\n\r\n";
unsafe {
libc::write(
client_fd,
request_bytes.as_ptr() as *const libc::c_void,
request_bytes.len(),
);
}
// Create connection on the "server" side.
// We need to manage the fd manually since Connection will close it on drop.
let dup_fd = unsafe { libc::dup(server_fd) };
let mut conn = Connection::new(dup_fd);
assert_eq!(conn.state, State::ReadingRequest);
// Read and parse.
assert!(conn.read().unwrap());
let req = conn.try_parse().unwrap();
assert_eq!(req.path, "/blog/test");
// Set response.
let response = Response::html("<h1>Test</h1>".into());
conn.set_response(&response);
assert_eq!(conn.state, State::WritingResponse);
// Write response.
let complete = conn.write().unwrap();
assert!(complete);
// Read from client side to verify.
let mut buf = [0u8; 4096];
let n = unsafe {
libc::read(client_fd, buf.as_mut_ptr() as *mut libc::c_void, buf.len())
};
assert!(n > 0);
let response_str = std::str::from_utf8(&buf[..n as usize]).unwrap();
assert!(response_str.contains("HTTP/1.1 200 OK"));
assert!(response_str.contains("<h1>Test</h1>"));
unsafe {
libc::close(server_fd);
libc::close(client_fd);
}
}
}

4
reference/crates/wo-http/src/lib.rs
View file

@ -1,4 +0,0 @@
pub mod listener;
pub mod request;
pub mod response;
pub mod connection;

202
reference/crates/wo-http/src/listener.rs
View file

@ -1,202 +0,0 @@
use std::io;
use std::os::unix::io::RawFd;
/// A non-blocking TCP listener wrapping raw socket syscalls.
pub struct TcpListener {
fd: RawFd,
}
impl TcpListener {
/// Bind to the given address (e.g., "0.0.0.0:3000") and start listening.
pub fn bind(addr: &str) -> io::Result<Self> {
let (ip, port) = parse_addr(addr)?;
let fd = unsafe {
libc::socket(
libc::AF_INET,
libc::SOCK_STREAM | libc::SOCK_NONBLOCK | libc::SOCK_CLOEXEC,
0,
)
};
if fd < 0 {
return Err(io::Error::last_os_error());
}
// SO_REUSEADDR to allow quick restart.
let optval: libc::c_int = 1;
let ret = unsafe {
libc::setsockopt(
fd,
libc::SOL_SOCKET,
libc::SO_REUSEADDR,
&optval as *const _ as *const libc::c_void,
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
)
};
if ret < 0 {
unsafe { libc::close(fd) };
return Err(io::Error::last_os_error());
}
let sockaddr = libc::sockaddr_in {
sin_family: libc::AF_INET as libc::sa_family_t,
sin_port: port.to_be(),
sin_addr: libc::in_addr { s_addr: ip },
sin_zero: [0; 8],
};
let ret = unsafe {
libc::bind(
fd,
&sockaddr as *const _ as *const libc::sockaddr,
std::mem::size_of::<libc::sockaddr_in>() as libc::socklen_t,
)
};
if ret < 0 {
unsafe { libc::close(fd) };
return Err(io::Error::last_os_error());
}
let ret = unsafe { libc::listen(fd, 128) };
if ret < 0 {
unsafe { libc::close(fd) };
return Err(io::Error::last_os_error());
}
Ok(Self { fd })
}
/// Accept a new connection. Returns the client fd or EAGAIN if none pending.
pub fn accept(&self) -> io::Result<Option<RawFd>> {
let client_fd = unsafe {
libc::accept4(
self.fd,
std::ptr::null_mut(),
std::ptr::null_mut(),
libc::SOCK_NONBLOCK | libc::SOCK_CLOEXEC,
)
};
if client_fd < 0 {
let err = io::Error::last_os_error();
if err.raw_os_error() == Some(libc::EAGAIN)
|| err.raw_os_error() == Some(libc::EWOULDBLOCK)
{
return Ok(None);
}
return Err(err);
}
Ok(Some(client_fd))
}
/// The listener file descriptor for epoll registration.
pub fn fd(&self) -> RawFd {
self.fd
}
/// Get the actual bound address (useful when binding to port 0).
pub fn local_addr(&self) -> io::Result<(String, u16)> {
let mut addr: libc::sockaddr_in = unsafe { std::mem::zeroed() };
let mut len = std::mem::size_of::<libc::sockaddr_in>() as libc::socklen_t;
let ret = unsafe {
libc::getsockname(
self.fd,
&mut addr as *mut _ as *mut libc::sockaddr,
&mut len,
)
};
if ret < 0 {
return Err(io::Error::last_os_error());
}
let ip = u32::from_be(addr.sin_addr.s_addr);
let port = u16::from_be(addr.sin_port);
let ip_str = format!(
"{}.{}.{}.{}",
(ip >> 24) & 0xFF,
(ip >> 16) & 0xFF,
(ip >> 8) & 0xFF,
ip & 0xFF,
);
Ok((ip_str, port))
}
}
impl Drop for TcpListener {
fn drop(&mut self) {
unsafe { libc::close(self.fd) };
}
}
fn parse_addr(addr: &str) -> io::Result<(u32, u16)> {
let parts: Vec<&str> = addr.rsplitn(2, ':').collect();
if parts.len() != 2 {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "expected host:port"));
}
let port: u16 = parts[0]
.parse()
.map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "invalid port"))?;
let ip_str = parts[1];
let ip = if ip_str == "0.0.0.0" {
0u32
} else if ip_str == "127.0.0.1" {
0x7F000001u32.to_be()
} else {
let octets: Vec<u8> = ip_str
.split('.')
.map(|s| s.parse().unwrap_or(0))
.collect();
if octets.len() != 4 {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "invalid IP"));
}
u32::from_ne_bytes([octets[0], octets[1], octets[2], octets[3]])
};
Ok((ip, port))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn bind_and_accept() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let (_, port) = listener.local_addr().unwrap();
assert!(port > 0);
// No pending connections → None.
assert!(listener.accept().unwrap().is_none());
// Connect a client.
let client_fd = unsafe {
libc::socket(libc::AF_INET, libc::SOCK_STREAM, 0)
};
assert!(client_fd >= 0);
let addr = libc::sockaddr_in {
sin_family: libc::AF_INET as libc::sa_family_t,
sin_port: port.to_be(),
sin_addr: libc::in_addr {
s_addr: 0x7F000001u32.to_be(),
},
sin_zero: [0; 8],
};
let ret = unsafe {
libc::connect(
client_fd,
&addr as *const _ as *const libc::sockaddr,
std::mem::size_of::<libc::sockaddr_in>() as libc::socklen_t,
)
};
assert_eq!(ret, 0);
// Now accept should return a fd.
let accepted = listener.accept().unwrap();
assert!(accepted.is_some());
unsafe {
libc::close(client_fd);
libc::close(accepted.unwrap());
}
}
}

158
reference/crates/wo-http/src/request.rs
View file

@ -1,158 +0,0 @@
use std::collections::HashMap;
/// HTTP method.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum Method {
Get,
Post,
Head,
Unknown,
}
/// A parsed HTTP/1.1 request.
#[derive(Debug, Clone)]
pub struct Request {
pub method: Method,
pub path: String,
pub query: Option<String>,
pub headers: HashMap<String, String>,
}
/// Result of attempting to parse a request from a byte buffer.
pub enum ParseResult {
/// Request fully parsed; returns the request and number of bytes consumed.
Complete(Request, usize),
/// Need more data.
Incomplete,
/// Malformed request.
Error(String),
}
/// Parse an HTTP/1.1 request from a byte buffer.
///
/// Returns `ParseResult::Complete` when the full header has been received
/// (delimited by `\r\n\r\n`).
pub fn parse(buf: &[u8]) -> ParseResult {
// Find the end of headers.
let header_end = match find_header_end(buf) {
Some(pos) => pos,
None => {
if buf.len() > 8192 {
return ParseResult::Error("request too large".into());
}
return ParseResult::Incomplete;
}
};
let header_str = match std::str::from_utf8(&buf[..header_end]) {
Ok(s) => s,
Err(_) => return ParseResult::Error("invalid UTF-8 in headers".into()),
};
let mut lines = header_str.lines();
// Request line: "GET /path HTTP/1.1"
let request_line = match lines.next() {
Some(l) => l,
None => return ParseResult::Error("empty request".into()),
};
let mut parts = request_line.split_whitespace();
let method = match parts.next() {
Some("GET") => Method::Get,
Some("POST") => Method::Post,
Some("HEAD") => Method::Head,
Some(_) => Method::Unknown,
None => return ParseResult::Error("missing method".into()),
};
let raw_path = match parts.next() {
Some(p) => p,
None => return ParseResult::Error("missing path".into()),
};
// Split path and query string.
let (path, query) = match raw_path.split_once('?') {
Some((p, q)) => (p.to_string(), Some(q.to_string())),
None => (raw_path.to_string(), None),
};
// Parse headers.
let mut headers = HashMap::new();
for line in lines {
if line.is_empty() {
break;
}
if let Some((key, value)) = line.split_once(':') {
headers.insert(
key.trim().to_lowercase(),
value.trim().to_string(),
);
}
}
// Bytes consumed: header + \r\n\r\n delimiter.
let consumed = header_end + 4;
ParseResult::Complete(Request { method, path, query, headers }, consumed)
}
fn find_header_end(buf: &[u8]) -> Option<usize> {
for i in 0..buf.len().saturating_sub(3) {
if &buf[i..i + 4] == b"\r\n\r\n" {
return Some(i);
}
}
None
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_simple_get() {
let raw = b"GET /blog/linux-misc HTTP/1.1\r\nHost: localhost\r\n\r\n";
match parse(raw) {
ParseResult::Complete(req, consumed) => {
assert_eq!(req.method, Method::Get);
assert_eq!(req.path, "/blog/linux-misc");
assert!(req.query.is_none());
assert_eq!(req.headers.get("host").unwrap(), "localhost");
assert_eq!(consumed, raw.len());
}
_ => panic!("expected Complete"),
}
}
#[test]
fn parse_with_query_string() {
let raw = b"GET /tag/rust?page=2 HTTP/1.1\r\n\r\n";
match parse(raw) {
ParseResult::Complete(req, _) => {
assert_eq!(req.path, "/tag/rust");
assert_eq!(req.query.as_deref(), Some("page=2"));
}
_ => panic!("expected Complete"),
}
}
#[test]
fn parse_incomplete() {
let raw = b"GET / HTTP/1.1\r\nHost: local";
assert!(matches!(parse(raw), ParseResult::Incomplete));
}
#[test]
fn parse_multiple_headers() {
let raw = b"GET / HTTP/1.1\r\nHost: localhost\r\nAccept: text/html\r\nConnection: keep-alive\r\n\r\n";
match parse(raw) {
ParseResult::Complete(req, _) => {
assert_eq!(req.headers.len(), 3);
assert_eq!(req.headers.get("accept").unwrap(), "text/html");
assert_eq!(req.headers.get("connection").unwrap(), "keep-alive");
}
_ => panic!("expected Complete"),
}
}
}

110
reference/crates/wo-http/src/response.rs
View file

@ -1,110 +0,0 @@
/// HTTP response builder.
#[derive(Debug, Clone)]
pub struct Response {
pub status: u16,
pub status_text: String,
pub headers: Vec<(String, String)>,
pub body: Vec<u8>,
}
impl Response {
pub fn new(status: u16, status_text: &str) -> Self {
Self {
status,
status_text: status_text.to_string(),
headers: vec![],
body: vec![],
}
}
/// 200 OK with HTML body.
pub fn html(body: String) -> Self {
let mut r = Self::new(200, "OK");
r.header("Content-Type", "text/html; charset=utf-8");
r.header("Content-Length", &body.len().to_string());
r.body = body.into_bytes();
r
}
/// 200 OK with a raw body and content type.
pub fn ok(body: Vec<u8>, content_type: &str) -> Self {
let mut r = Self::new(200, "OK");
r.header("Content-Type", content_type);
r.header("Content-Length", &body.len().to_string());
r.body = body;
r
}
/// 404 Not Found.
pub fn not_found() -> Self {
let body = "<h1>404 Not Found</h1>";
let mut r = Self::new(404, "Not Found");
r.header("Content-Type", "text/html; charset=utf-8");
r.header("Content-Length", &body.len().to_string());
r.body = body.as_bytes().to_vec();
r
}
/// 500 Internal Server Error.
pub fn internal_error(msg: &str) -> Self {
let body = format!("<h1>500 Internal Server Error</h1><p>{}</p>", msg);
let mut r = Self::new(500, "Internal Server Error");
r.header("Content-Type", "text/html; charset=utf-8");
r.header("Content-Length", &body.len().to_string());
r.body = body.into_bytes();
r
}
/// Add a header.
pub fn header(&mut self, key: &str, value: &str) -> &mut Self {
self.headers.push((key.to_string(), value.to_string()));
self
}
/// Serialize the response to bytes for writing to a socket.
pub fn to_bytes(&self) -> Vec<u8> {
let mut buf = Vec::with_capacity(256 + self.body.len());
// Status line.
buf.extend_from_slice(
format!("HTTP/1.1 {} {}\r\n", self.status, self.status_text).as_bytes(),
);
// Headers.
for (key, value) in &self.headers {
buf.extend_from_slice(format!("{}: {}\r\n", key, value).as_bytes());
}
// End of headers.
buf.extend_from_slice(b"\r\n");
// Body.
buf.extend_from_slice(&self.body);
buf
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn html_response() {
let r = Response::html("<h1>Hello</h1>".into());
let bytes = r.to_bytes();
let s = String::from_utf8(bytes).unwrap();
assert!(s.starts_with("HTTP/1.1 200 OK\r\n"));
assert!(s.contains("Content-Type: text/html"));
assert!(s.contains("<h1>Hello</h1>"));
}
#[test]
fn not_found_response() {
let r = Response::not_found();
assert_eq!(r.status, 404);
let bytes = r.to_bytes();
let s = String::from_utf8(bytes).unwrap();
assert!(s.contains("404 Not Found"));
}
}

13
reference/crates/wo-index/Cargo.toml
View file

@ -1,13 +0,0 @@
[package]
name = "wo-index"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-model = { path = "../wo-model" }
libc = { workspace = true }
bincode = { workspace = true }
serde = { workspace = true }
[dev-dependencies]
tempfile = "3"

194
reference/crates/wo-index/src/date.rs
View file

@ -1,194 +0,0 @@
use std::io;
use std::path::Path;
/// Sorted index mapping timestamps to byte offsets in a .seg file.
///
/// On-disk format:
/// ```text
/// [8 bytes] entry_count: u64 LE
/// For each entry (sorted by timestamp ascending):
/// [8 bytes] timestamp: i64 LE
/// [8 bytes] offset: u64 LE
/// ```
const ENTRY_SIZE: usize = 16; // i64 + u64
const HEADER_SIZE: usize = 8;
pub struct DateIndex {
data: Vec<u8>,
count: usize,
}
impl DateIndex {
/// Build a date index. Entries are sorted by timestamp before writing.
pub fn build(path: &Path, entries: &mut [(i64, u64)]) -> io::Result<()> {
entries.sort_by_key(|e| e.0);
let file_size = HEADER_SIZE + entries.len() * ENTRY_SIZE;
let mut data = vec![0u8; file_size];
data[0..8].copy_from_slice(&(entries.len() as u64).to_le_bytes());
for (i, &(ts, offset)) in entries.iter().enumerate() {
let base = HEADER_SIZE + i * ENTRY_SIZE;
data[base..base + 8].copy_from_slice(&ts.to_le_bytes());
data[base + 8..base + 16].copy_from_slice(&offset.to_le_bytes());
}
std::fs::write(path, &data)?;
Ok(())
}
/// Open a date index from disk.
pub fn open(path: &Path) -> io::Result<Self> {
let data = std::fs::read(path)?;
if data.len() < HEADER_SIZE {
return Err(io::Error::new(io::ErrorKind::InvalidData, "date.idx too small"));
}
let count = u64::from_le_bytes(data[0..8].try_into().unwrap()) as usize;
Ok(Self { data, count })
}
/// Get offsets for all entries with timestamps in [start, end].
pub fn range(&self, start: i64, end: i64) -> Vec<u64> {
let lo = self.lower_bound(start);
let hi = self.upper_bound(end);
(lo..hi).map(|i| self.offset_at(i)).collect()
}
/// Get offsets for the N most recent entries (highest timestamps).
pub fn latest(&self, n: usize) -> Vec<u64> {
let start = self.count.saturating_sub(n);
(start..self.count)
.rev()
.map(|i| self.offset_at(i))
.collect()
}
/// Number of entries.
pub fn len(&self) -> usize {
self.count
}
pub fn is_empty(&self) -> bool {
self.count == 0
}
fn timestamp_at(&self, i: usize) -> i64 {
let base = HEADER_SIZE + i * ENTRY_SIZE;
i64::from_le_bytes(self.data[base..base + 8].try_into().unwrap())
}
fn offset_at(&self, i: usize) -> u64 {
let base = HEADER_SIZE + i * ENTRY_SIZE;
u64::from_le_bytes(self.data[base + 8..base + 16].try_into().unwrap())
}
/// Binary search: first index where timestamp >= target.
fn lower_bound(&self, target: i64) -> usize {
let (mut lo, mut hi) = (0usize, self.count);
while lo < hi {
let mid = lo + (hi - lo) / 2;
if self.timestamp_at(mid) < target {
lo = mid + 1;
} else {
hi = mid;
}
}
lo
}
/// Binary search: first index where timestamp > target.
fn upper_bound(&self, target: i64) -> usize {
let (mut lo, mut hi) = (0usize, self.count);
while lo < hi {
let mid = lo + (hi - lo) / 2;
if self.timestamp_at(mid) <= target {
lo = mid + 1;
} else {
hi = mid;
}
}
lo
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn build_and_range_query() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("date.idx");
let mut entries = vec![
(1000i64, 100u64),
(2000, 200),
(3000, 300),
(4000, 400),
(5000, 500),
];
DateIndex::build(&path, &mut entries).unwrap();
let idx = DateIndex::open(&path).unwrap();
assert_eq!(idx.len(), 5);
// Full range.
let all = idx.range(0, 9999);
assert_eq!(all, vec![100, 200, 300, 400, 500]);
// Partial range.
let mid = idx.range(2000, 4000);
assert_eq!(mid, vec![200, 300, 400]);
// Single.
let one = idx.range(3000, 3000);
assert_eq!(one, vec![300]);
// Empty range.
let none = idx.range(6000, 9000);
assert!(none.is_empty());
}
#[test]
fn latest_entries() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("date.idx");
let mut entries = vec![
(1000i64, 100u64),
(2000, 200),
(3000, 300),
];
DateIndex::build(&path, &mut entries).unwrap();
let idx = DateIndex::open(&path).unwrap();
let top2 = idx.latest(2);
assert_eq!(top2, vec![300, 200]);
let top10 = idx.latest(10);
assert_eq!(top10, vec![300, 200, 100]);
}
#[test]
fn unsorted_input() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("date.idx");
let mut entries = vec![
(5000i64, 500u64),
(1000, 100),
(3000, 300),
];
DateIndex::build(&path, &mut entries).unwrap();
let idx = DateIndex::open(&path).unwrap();
// Should be sorted on disk.
let all = idx.range(0, 9999);
assert_eq!(all, vec![100, 300, 500]);
}
}

40
reference/crates/wo-index/src/lib.rs
View file

@ -1,40 +0,0 @@
mod title;
mod date;
mod tags;
pub use title::TitleIndex;
pub use date::DateIndex;
pub use tags::TagIndex;
use std::io;
use std::path::Path;
/// Rebuild all three index files from an iterator of article metadata.
///
/// Each entry is `(sys_title, published_on_timestamp, tags, seg_offset)`.
pub fn rebuild_all(
index_dir: &Path,
entries: &[(String, i64, Vec<String>, u64)],
) -> io::Result<()> {
std::fs::create_dir_all(index_dir)?;
// Build title index.
let title_entries: Vec<(&str, u64)> = entries.iter().map(|e| (e.0.as_str(), e.3)).collect();
TitleIndex::build(&index_dir.join("title.idx"), &title_entries)?;
// Build date index.
let mut date_entries: Vec<(i64, u64)> = entries.iter().map(|e| (e.1, e.3)).collect();
DateIndex::build(&index_dir.join("date.idx"), &mut date_entries)?;
// Build tags index.
let mut tag_map: std::collections::HashMap<String, Vec<u64>> = std::collections::HashMap::new();
for entry in entries {
for tag in &entry.2 {
tag_map.entry(tag.clone()).or_default().push(entry.3);
}
}
let tag_entries: Vec<(String, Vec<u64>)> = tag_map.into_iter().collect();
TagIndex::build(&index_dir.join("tags.idx"), &tag_entries)?;
Ok(())
}

97
reference/crates/wo-index/src/tags.rs
View file

@ -1,97 +0,0 @@
use std::collections::HashMap;
use std::io;
use std::path::Path;
use serde::{Deserialize, Serialize};
/// Inverted index mapping tag strings to lists of .seg byte offsets.
///
/// Serialized to disk using bincode for simplicity (article count is
/// small enough that loading the full index into memory is fine).
#[derive(Debug, Serialize, Deserialize)]
struct TagIndexData {
tags: HashMap<String, Vec<u64>>,
}
pub struct TagIndex {
tags: HashMap<String, Vec<u64>>,
}
impl TagIndex {
/// Build a tags index from a list of (tag, offsets) pairs.
pub fn build(path: &Path, entries: &[(String, Vec<u64>)]) -> io::Result<()> {
let tags: HashMap<String, Vec<u64>> = entries.iter().cloned().collect();
let data = TagIndexData { tags };
let bytes = bincode::serialize(&data).map_err(|e| {
io::Error::new(io::ErrorKind::InvalidData, format!("bincode: {}", e))
})?;
std::fs::write(path, bytes)?;
Ok(())
}
/// Open a tags index from disk.
pub fn open(path: &Path) -> io::Result<Self> {
let bytes = std::fs::read(path)?;
let data: TagIndexData = bincode::deserialize(&bytes).map_err(|e| {
io::Error::new(io::ErrorKind::InvalidData, format!("bincode: {}", e))
})?;
Ok(Self { tags: data.tags })
}
/// Get all offsets for articles tagged with the given tag.
pub fn get(&self, tag: &str) -> Option<&[u64]> {
self.tags.get(tag).map(|v| v.as_slice())
}
/// All tag names in the index.
pub fn tags(&self) -> Vec<&str> {
self.tags.keys().map(|s| s.as_str()).collect()
}
/// Number of distinct tags.
pub fn len(&self) -> usize {
self.tags.len()
}
pub fn is_empty(&self) -> bool {
self.tags.is_empty()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn build_and_lookup() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("tags.idx");
let entries = vec![
("rust".to_string(), vec![100u64, 200]),
("linux".to_string(), vec![300]),
("tutorial".to_string(), vec![100, 300, 400]),
];
TagIndex::build(&path, &entries).unwrap();
let idx = TagIndex::open(&path).unwrap();
assert_eq!(idx.len(), 3);
assert_eq!(idx.get("rust"), Some(vec![100u64, 200].as_slice()));
assert_eq!(idx.get("linux"), Some(vec![300u64].as_slice()));
assert_eq!(idx.get("tutorial"), Some(vec![100u64, 300, 400].as_slice()));
assert_eq!(idx.get("nonexistent"), None);
}
#[test]
fn empty_index() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("tags.idx");
TagIndex::build(&path, &[]).unwrap();
let idx = TagIndex::open(&path).unwrap();
assert!(idx.is_empty());
assert_eq!(idx.get("anything"), None);
}
}

183
reference/crates/wo-index/src/title.rs
View file

@ -1,183 +0,0 @@
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
use std::io;
use std::path::Path;
/// On-disk hash index mapping `sys_title` (string) to byte offset in a .seg file.
///
/// Uses open addressing with linear probing. The table is sized at 2x the entry
/// count (load factor 0.5) to keep collision chains short.
///
/// On-disk format:
/// ```text
/// [8 bytes] table_size: u64 LE (number of slots)
/// [8 bytes] entry_count: u64 LE
/// For each slot (table_size slots):
/// [8 bytes] offset: u64 LE (0 = empty, otherwise seg offset + 1)
/// [2 bytes] key_len: u16 LE
/// [128 bytes] key_data: zero-padded sys_title
/// Total slot size: 138 bytes
/// ```
const SLOT_SIZE: usize = 8 + 2 + 128; // offset + key_len + key_data
const HEADER_SIZE: usize = 16; // table_size + entry_count
const MAX_KEY_LEN: usize = 128;
pub struct TitleIndex {
data: Vec<u8>,
table_size: u64,
}
impl TitleIndex {
/// Build a title index file from a list of (sys_title, seg_offset) pairs.
pub fn build(path: &Path, entries: &[(&str, u64)]) -> io::Result<()> {
let table_size = (entries.len() * 2).max(16) as u64;
let file_size = HEADER_SIZE + (table_size as usize) * SLOT_SIZE;
let mut data = vec![0u8; file_size];
// Write header.
data[0..8].copy_from_slice(&table_size.to_le_bytes());
data[8..16].copy_from_slice(&(entries.len() as u64).to_le_bytes());
// Insert entries.
for &(key, offset) in entries {
let slot = Self::find_empty_slot(&data, table_size, key);
Self::write_slot(&mut data, slot, key, offset);
}
std::fs::write(path, &data)?;
Ok(())
}
/// Open a title index from disk (loads into memory).
pub fn open(path: &Path) -> io::Result<Self> {
let data = std::fs::read(path)?;
if data.len() < HEADER_SIZE {
return Err(io::Error::new(io::ErrorKind::InvalidData, "title.idx too small"));
}
let table_size = u64::from_le_bytes(data[0..8].try_into().unwrap());
Ok(Self { data, table_size })
}
/// Look up a sys_title, returning the seg file byte offset if found.
pub fn get(&self, sys_title: &str) -> Option<u64> {
let mut slot = Self::hash_key(sys_title) % self.table_size;
for _ in 0..self.table_size {
let (stored_offset, stored_key) = self.read_slot(slot as usize);
if stored_offset == 0 {
return None; // empty slot, key not found
}
if stored_key == sys_title {
return Some(stored_offset - 1); // stored as offset + 1
}
slot = (slot + 1) % self.table_size;
}
None
}
/// Number of entries in the index.
pub fn len(&self) -> usize {
u64::from_le_bytes(self.data[8..16].try_into().unwrap()) as usize
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
fn hash_key(key: &str) -> u64 {
let mut hasher = DefaultHasher::new();
key.hash(&mut hasher);
hasher.finish()
}
fn find_empty_slot(data: &[u8], table_size: u64, key: &str) -> usize {
let mut slot = (Self::hash_key(key) % table_size) as usize;
loop {
let base = HEADER_SIZE + slot * SLOT_SIZE;
let stored_offset = u64::from_le_bytes(data[base..base + 8].try_into().unwrap());
if stored_offset == 0 {
return slot;
}
slot = (slot + 1) % table_size as usize;
}
}
fn write_slot(data: &mut [u8], slot: usize, key: &str, offset: u64) {
let base = HEADER_SIZE + slot * SLOT_SIZE;
let stored_offset = offset + 1; // +1 so that 0 means empty
data[base..base + 8].copy_from_slice(&stored_offset.to_le_bytes());
let key_bytes = key.as_bytes();
let key_len = key_bytes.len().min(MAX_KEY_LEN) as u16;
data[base + 8..base + 10].copy_from_slice(&key_len.to_le_bytes());
data[base + 10..base + 10 + key_len as usize].copy_from_slice(&key_bytes[..key_len as usize]);
}
fn read_slot(&self, slot: usize) -> (u64, String) {
let base = HEADER_SIZE + slot * SLOT_SIZE;
let stored_offset = u64::from_le_bytes(self.data[base..base + 8].try_into().unwrap());
let key_len = u16::from_le_bytes(self.data[base + 8..base + 10].try_into().unwrap()) as usize;
let key = String::from_utf8_lossy(&self.data[base + 10..base + 10 + key_len]).to_string();
(stored_offset, key)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn build_and_lookup() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("title.idx");
let entries = vec![
("linux-misc", 32u64),
("aws-lambda-pulumi", 500),
("rust-patterns", 1200),
];
TitleIndex::build(&path, &entries).unwrap();
let idx = TitleIndex::open(&path).unwrap();
assert_eq!(idx.len(), 3);
assert_eq!(idx.get("linux-misc"), Some(32));
assert_eq!(idx.get("aws-lambda-pulumi"), Some(500));
assert_eq!(idx.get("rust-patterns"), Some(1200));
assert_eq!(idx.get("nonexistent"), None);
}
#[test]
fn empty_index() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("title.idx");
TitleIndex::build(&path, &[]).unwrap();
let idx = TitleIndex::open(&path).unwrap();
assert!(idx.is_empty());
assert_eq!(idx.get("anything"), None);
}
#[test]
fn many_entries() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("title.idx");
let keys: Vec<String> = (0..100).map(|i| format!("article-{}", i)).collect();
let entries: Vec<(&str, u64)> = keys.iter().enumerate().map(|(i, k)| (k.as_str(), i as u64 * 100)).collect();
TitleIndex::build(&path, &entries).unwrap();
let idx = TitleIndex::open(&path).unwrap();
assert_eq!(idx.len(), 100);
for (i, key) in keys.iter().enumerate() {
assert_eq!(idx.get(key), Some(i as u64 * 100));
}
}
}

6
reference/crates/wo-md/Cargo.toml
View file

@ -1,6 +0,0 @@
[package]
name = "wo-md"
version = "0.1.0"
edition = "2021"
[dependencies]

324
reference/crates/wo-md/src/block.rs
View file

@ -1,324 +0,0 @@
use crate::highlight::highlight;
use crate::inline::markdown_to_html;
/// Parse a full markdown document into HTML.
///
/// Handles block-level elements (headings, paragraphs, code fences, lists,
/// blockquotes, images) and delegates inline formatting to `markdown_to_html`.
pub fn markdown_to_html_block(md: &str) -> String {
let lines: Vec<&str> = md.lines().collect();
let len = lines.len();
let mut html = String::with_capacity(md.len() * 2);
let mut i = 0;
while i < len {
let line = lines[i];
let trimmed = line.trim();
// Skip blank lines.
if trimmed.is_empty() {
i += 1;
continue;
}
// Fenced code block: ```lang ... ```
if trimmed.starts_with("```") {
let lang = trimmed[3..].trim();
i += 1;
let mut code = String::new();
while i < len && !lines[i].trim().starts_with("```") {
if !code.is_empty() {
code.push('\n');
}
code.push_str(lines[i]);
i += 1;
}
if i < len {
i += 1; // skip closing ```
}
if lang.is_empty() {
html.push_str("<pre><code>");
html.push_str(&escape_html(&code));
html.push_str("</code></pre>\n");
} else {
html.push_str(&format!("<pre><code class=\"language-{}\">", escape_html(lang)));
html.push_str(&highlight(&code, lang));
html.push_str("</code></pre>\n");
}
continue;
}
// Heading: # ... ######
if trimmed.starts_with('#') {
let level = trimmed.chars().take_while(|&c| c == '#').count().min(6);
let text = trimmed[level..].trim();
html.push_str(&format!(
"<h{0}>{1}</h{0}>\n",
level,
markdown_to_html(text)
));
i += 1;
continue;
}
// Blockquote: > text
if trimmed.starts_with('>') {
let mut quote_lines = Vec::new();
while i < len && lines[i].trim().starts_with('>') {
let content = lines[i].trim().strip_prefix('>').unwrap_or("").trim();
quote_lines.push(content);
i += 1;
}
html.push_str("<blockquote>");
html.push_str(&markdown_to_html(&quote_lines.join(" ")));
html.push_str("</blockquote>\n");
continue;
}
// Unordered list: - item or * item
if (trimmed.starts_with("- ") || trimmed.starts_with("* "))
&& !trimmed.starts_with("---")
{
html.push_str("<ul>\n");
while i < len {
let lt = lines[i].trim();
if lt.starts_with("- ") || lt.starts_with("* ") {
let text = &lt[2..];
html.push_str(&format!("<li>{}</li>\n", markdown_to_html(text)));
i += 1;
} else if lt.is_empty() {
i += 1;
break;
} else {
break;
}
}
html.push_str("</ul>\n");
continue;
}
// Ordered list: 1. item
if trimmed.len() > 2 && trimmed.as_bytes()[0].is_ascii_digit() {
if let Some(rest) = strip_ordered_prefix(trimmed) {
html.push_str("<ol>\n");
html.push_str(&format!("<li>{}</li>\n", markdown_to_html(rest)));
i += 1;
while i < len {
let lt = lines[i].trim();
if let Some(rest) = strip_ordered_prefix(lt) {
html.push_str(&format!("<li>{}</li>\n", markdown_to_html(rest)));
i += 1;
} else if lt.is_empty() {
i += 1;
break;
} else {
break;
}
}
html.push_str("</ol>\n");
continue;
}
}
// Image: ![alt](url)
if trimmed.starts_with("![") {
if let Some((alt, url)) = parse_image(trimmed) {
html.push_str(&format!(
"<img src=\"{}\" alt=\"{}\">\n",
escape_html(url),
escape_html(alt)
));
i += 1;
continue;
}
}
// Horizontal rule: --- or ***
if trimmed == "---" || trimmed == "***" || trimmed == "___" {
html.push_str("<hr>\n");
i += 1;
continue;
}
// Paragraph: collect consecutive non-blank, non-block lines.
let mut para_lines = Vec::new();
while i < len {
let lt = lines[i].trim();
if lt.is_empty()
|| lt.starts_with('#')
|| lt.starts_with("```")
|| lt.starts_with('>')
|| lt == "---"
|| lt == "***"
|| lt == "___"
{
break;
}
// Check if next line starts a list
if (lt.starts_with("- ") || lt.starts_with("* ")) && !lt.starts_with("---") {
break;
}
if strip_ordered_prefix(lt).is_some() && para_lines.is_empty() {
break;
}
para_lines.push(lt);
i += 1;
}
if !para_lines.is_empty() {
html.push_str("<p>");
html.push_str(&markdown_to_html(&para_lines.join(" ")));
html.push_str("</p>\n");
}
}
html
}
fn escape_html(s: &str) -> String {
s.replace('&', "&amp;")
.replace('<', "&lt;")
.replace('>', "&gt;")
.replace('"', "&quot;")
}
fn strip_ordered_prefix(s: &str) -> Option<&str> {
let dot_pos = s.find(". ")?;
if dot_pos > 0 && s[..dot_pos].chars().all(|c| c.is_ascii_digit()) {
Some(&s[dot_pos + 2..])
} else {
None
}
}
fn parse_image(s: &str) -> Option<(&str, &str)> {
// ![alt](url)
let alt_start = s.find("![")? + 2;
let alt_end = s[alt_start..].find(']')? + alt_start;
let url_start = s[alt_end..].find('(')? + alt_end + 1;
let url_end = s[url_start..].find(')')? + url_start;
Some((&s[alt_start..alt_end], &s[url_start..url_end]))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn headings() {
let md = "# Title\n\n## Subtitle\n\n### Third";
let html = markdown_to_html_block(md);
assert!(html.contains("<h1>Title</h1>"));
assert!(html.contains("<h2>Subtitle</h2>"));
assert!(html.contains("<h3>Third</h3>"));
}
#[test]
fn paragraphs() {
let md = "First paragraph.\n\nSecond paragraph.";
let html = markdown_to_html_block(md);
assert!(html.contains("<p>First paragraph.</p>"));
assert!(html.contains("<p>Second paragraph.</p>"));
}
#[test]
fn code_fence_with_language() {
let md = "```rust\nfn main() {\n let x = 42;\n}\n```";
let html = markdown_to_html_block(md);
assert!(html.contains("<pre><code class=\"language-rust\">"));
assert!(html.contains("<span class=\"kw\">fn</span>"));
assert!(html.contains("<span class=\"num\">42</span>"));
assert!(html.contains("</code></pre>"));
}
#[test]
fn code_fence_no_language() {
let md = "```\nplain code\n```";
let html = markdown_to_html_block(md);
assert!(html.contains("<pre><code>plain code</code></pre>"));
}
#[test]
fn unordered_list() {
let md = "- Item one\n- Item two\n- Item three";
let html = markdown_to_html_block(md);
assert!(html.contains("<ul>"));
assert!(html.contains("<li>Item one</li>"));
assert!(html.contains("<li>Item two</li>"));
assert!(html.contains("</ul>"));
}
#[test]
fn ordered_list() {
let md = "1. First\n2. Second\n3. Third";
let html = markdown_to_html_block(md);
assert!(html.contains("<ol>"));
assert!(html.contains("<li>First</li>"));
assert!(html.contains("<li>Third</li>"));
assert!(html.contains("</ol>"));
}
#[test]
fn blockquote() {
let md = "> This is a quote\n> spanning two lines";
let html = markdown_to_html_block(md);
assert!(html.contains("<blockquote>This is a quote spanning two lines</blockquote>"));
}
#[test]
fn image() {
let md = "![Alt text](https://example.com/img.png)";
let html = markdown_to_html_block(md);
assert!(html.contains("<img src=\"https://example.com/img.png\" alt=\"Alt text\">"));
}
#[test]
fn horizontal_rule() {
let md = "Before\n\n---\n\nAfter";
let html = markdown_to_html_block(md);
assert!(html.contains("<hr>"));
}
#[test]
fn inline_formatting_in_paragraphs() {
let md = "Use **bold** and `code` in a paragraph.";
let html = markdown_to_html_block(md);
assert!(html.contains("<strong>bold</strong>"));
assert!(html.contains("<code>code</code>"));
}
#[test]
fn full_article() {
let md = "\
# Getting Started
Welcome to the guide.
## Installation
Install with cargo:
```bash
cargo install writeonce
```
## Features
- Fast rendering
- Zero dependencies
- **Bold** feature
> Note: this is a quote.
![Logo](logo.png)
";
let html = markdown_to_html_block(md);
assert!(html.contains("<h1>Getting Started</h1>"));
assert!(html.contains("<h2>Installation</h2>"));
assert!(html.contains("<p>Install with cargo:</p>"));
assert!(html.contains("<pre><code class=\"language-bash\">"));
assert!(html.contains("<li>Fast rendering</li>"));
assert!(html.contains("<blockquote>"));
assert!(html.contains("<img src=\"logo.png\""));
}
}

154
reference/crates/wo-md/src/highlight.rs
View file

@ -1,154 +0,0 @@
/// Server-side syntax highlighting for code blocks.
///
/// Returns HTML with `<span class="...">` tokens. Pair with code-theme.css.
pub fn highlight(code: &str, language: &str) -> String {
let keywords = keywords_for(language);
let mut result = String::with_capacity(code.len() * 2);
let chars: Vec<char> = code.chars().collect();
let len = chars.len();
let mut i = 0;
while i < len {
// String literals (double or single quotes).
if chars[i] == '"' || chars[i] == '\'' {
let quote = chars[i];
let start = i;
i += 1;
while i < len && chars[i] != quote {
if chars[i] == '\\' { i += 1; } // skip escaped char
i += 1;
}
if i < len { i += 1; } // closing quote
let s: String = chars[start..i].iter().collect();
result.push_str(&format!("<span class=\"str\">{}</span>", escape(&s)));
continue;
}
// Line comments.
if i + 1 < len && chars[i] == '/' && chars[i + 1] == '/' {
let start = i;
while i < len && chars[i] != '\n' { i += 1; }
let s: String = chars[start..i].iter().collect();
result.push_str(&format!("<span class=\"cm\">{}</span>", escape(&s)));
continue;
}
// Hash comments (bash, yaml, etc.).
if chars[i] == '#' && (language == "bash" || language == "yaml" || language == "python") {
let start = i;
while i < len && chars[i] != '\n' { i += 1; }
let s: String = chars[start..i].iter().collect();
result.push_str(&format!("<span class=\"cm\">{}</span>", escape(&s)));
continue;
}
// Numbers.
if chars[i].is_ascii_digit() {
let start = i;
while i < len && (chars[i].is_ascii_alphanumeric() || chars[i] == '.') { i += 1; }
let s: String = chars[start..i].iter().collect();
result.push_str(&format!("<span class=\"num\">{}</span>", escape(&s)));
continue;
}
// Identifiers / keywords.
if chars[i].is_ascii_alphabetic() || chars[i] == '_' {
let start = i;
while i < len && (chars[i].is_ascii_alphanumeric() || chars[i] == '_') { i += 1; }
let word: String = chars[start..i].iter().collect();
if keywords.contains(&word.as_str()) {
result.push_str(&format!("<span class=\"kw\">{}</span>", escape(&word)));
} else {
result.push_str(&escape(&word));
}
continue;
}
// Everything else.
result.push_str(&escape(&chars[i].to_string()));
i += 1;
}
result
}
fn escape(s: &str) -> String {
s.replace('&', "&amp;")
.replace('<', "&lt;")
.replace('>', "&gt;")
}
fn keywords_for(language: &str) -> &'static [&'static str] {
match language {
"rust" => &[
"fn", "let", "mut", "const", "struct", "enum", "impl", "trait", "pub", "use",
"mod", "crate", "self", "super", "return", "if", "else", "match", "for", "while",
"loop", "break", "continue", "where", "type", "as", "in", "ref", "move",
"async", "await", "unsafe", "extern", "dyn", "true", "false",
],
"go" => &[
"func", "var", "const", "type", "struct", "interface", "map", "chan",
"package", "import", "return", "if", "else", "for", "range", "switch",
"case", "default", "go", "defer", "select", "true", "false", "nil",
],
"bash" | "sh" => &[
"if", "then", "else", "elif", "fi", "for", "while", "do", "done",
"case", "esac", "function", "return", "export", "local", "echo",
"in", "true", "false",
],
"yaml" => &["true", "false", "null", "yes", "no"],
"json" => &["true", "false", "null"],
"python" => &[
"def", "class", "if", "elif", "else", "for", "while", "return",
"import", "from", "as", "with", "try", "except", "finally",
"raise", "pass", "lambda", "yield", "True", "False", "None", "in",
"not", "and", "or", "is", "async", "await",
],
_ => &[],
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn highlight_rust() {
let code = "fn main() {\n let x = 42;\n}";
let html = highlight(code, "rust");
assert!(html.contains("<span class=\"kw\">fn</span>"));
assert!(html.contains("<span class=\"kw\">let</span>"));
assert!(html.contains("<span class=\"num\">42</span>"));
}
#[test]
fn highlight_string() {
let code = r#"let s = "hello";"#;
let html = highlight(code, "rust");
assert!(html.contains("<span class=\"str\">\"hello\"</span>"));
}
#[test]
fn highlight_comment() {
let code = "// this is a comment\nlet x = 1;";
let html = highlight(code, "rust");
assert!(html.contains("<span class=\"cm\">// this is a comment</span>"));
}
#[test]
fn highlight_bash() {
let code = "# comment\nexport PATH=/usr/bin";
let html = highlight(code, "bash");
assert!(html.contains("<span class=\"cm\"># comment</span>"));
assert!(html.contains("<span class=\"kw\">export</span>"));
}
#[test]
fn escapes_html() {
let code = "fn compare<T>(a: T, b: T) {}";
let html = highlight(code, "rust");
assert!(html.contains("&lt;"));
assert!(html.contains("&gt;"));
assert!(!html.contains("<T>"));
}
}

171
reference/crates/wo-md/src/inline.rs
View file

@ -1,171 +0,0 @@
/// Convert inline markdown to HTML.
///
/// Handles: **bold**, *italic*, `code`, [text](url), and HTML entity escaping.
/// Does not handle block-level elements (headers, lists, etc.) — those are
/// already structured in the article JSON.
pub fn markdown_to_html(text: &str) -> String {
let mut result = String::with_capacity(text.len() * 2);
let chars: Vec<char> = text.chars().collect();
let len = chars.len();
let mut i = 0;
while i < len {
match chars[i] {
// HTML entity escaping.
'&' => { result.push_str("&amp;"); i += 1; }
'<' => { result.push_str("&lt;"); i += 1; }
'>' => { result.push_str("&gt;"); i += 1; }
// **bold** or *italic*
'*' => {
if i + 1 < len && chars[i + 1] == '*' {
// **bold**
if let Some(end) = find_closing(&chars, i + 2, "**") {
result.push_str("<strong>");
let inner: String = chars[i + 2..end].iter().collect();
result.push_str(&escape_html(&inner));
result.push_str("</strong>");
i = end + 2;
} else {
result.push('*');
i += 1;
}
} else {
// *italic*
if let Some(end) = find_closing_char(&chars, i + 1, '*') {
result.push_str("<em>");
let inner: String = chars[i + 1..end].iter().collect();
result.push_str(&escape_html(&inner));
result.push_str("</em>");
i = end + 1;
} else {
result.push('*');
i += 1;
}
}
}
// `inline code`
'`' => {
if let Some(end) = find_closing_char(&chars, i + 1, '`') {
result.push_str("<code>");
let inner: String = chars[i + 1..end].iter().collect();
result.push_str(&escape_html(&inner));
result.push_str("</code>");
i = end + 1;
} else {
result.push('`');
i += 1;
}
}
// [text](url)
'[' => {
if let Some((text_end, url_start, url_end)) = parse_link(&chars, i) {
let link_text: String = chars[i + 1..text_end].iter().collect();
let url: String = chars[url_start..url_end].iter().collect();
result.push_str(&format!(
"<a href=\"{}\">{}</a>",
escape_html(&url),
escape_html(&link_text)
));
i = url_end + 1;
} else {
result.push('[');
i += 1;
}
}
c => {
result.push(c);
i += 1;
}
}
}
result
}
fn escape_html(s: &str) -> String {
s.replace('&', "&amp;")
.replace('<', "&lt;")
.replace('>', "&gt;")
.replace('"', "&quot;")
}
fn find_closing(chars: &[char], start: usize, marker: &str) -> Option<usize> {
let marker_chars: Vec<char> = marker.chars().collect();
let mlen = marker_chars.len();
for i in start..chars.len().saturating_sub(mlen - 1) {
if chars[i..i + mlen] == marker_chars[..] {
return Some(i);
}
}
None
}
fn find_closing_char(chars: &[char], start: usize, marker: char) -> Option<usize> {
for i in start..chars.len() {
if chars[i] == marker {
return Some(i);
}
}
None
}
fn parse_link(chars: &[char], start: usize) -> Option<(usize, usize, usize)> {
// [text](url)
let text_end = find_closing_char(chars, start + 1, ']')?;
if text_end + 1 >= chars.len() || chars[text_end + 1] != '(' {
return None;
}
let url_start = text_end + 2;
let url_end = find_closing_char(chars, url_start, ')')?;
Some((text_end, url_start, url_end))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn plain_text() {
assert_eq!(markdown_to_html("hello world"), "hello world");
}
#[test]
fn bold() {
assert_eq!(markdown_to_html("**bold**"), "<strong>bold</strong>");
}
#[test]
fn italic() {
assert_eq!(markdown_to_html("*italic*"), "<em>italic</em>");
}
#[test]
fn inline_code() {
assert_eq!(markdown_to_html("`code`"), "<code>code</code>");
}
#[test]
fn link() {
assert_eq!(
markdown_to_html("[click](https://example.com)"),
"<a href=\"https://example.com\">click</a>"
);
}
#[test]
fn mixed() {
assert_eq!(
markdown_to_html("Use **Rust** with `cargo` for [docs](https://doc.rust-lang.org)"),
"Use <strong>Rust</strong> with <code>cargo</code> for <a href=\"https://doc.rust-lang.org\">docs</a>"
);
}
#[test]
fn html_escaping() {
assert_eq!(markdown_to_html("<script>alert('xss')</script>"), "&lt;script&gt;alert('xss')&lt;/script&gt;");
}
}

7
reference/crates/wo-md/src/lib.rs
View file

@ -1,7 +0,0 @@
mod block;
mod inline;
mod highlight;
pub use block::markdown_to_html_block;
pub use inline::markdown_to_html;
pub use highlight::highlight;

13
reference/crates/wo-model/Cargo.toml
View file

@ -1,13 +0,0 @@
[package]
name = "wo-model"
version = "0.1.0"
edition = "2021"
[dependencies]
serde = { workspace = true }
serde_json = { workspace = true }
bincode = { workspace = true }
wo-md = { path = "../wo-md" }
[dev-dependencies]
tempfile = "3"

177
reference/crates/wo-model/src/article.rs
View file

@ -1,177 +0,0 @@
use serde::{Deserialize, Serialize};
/// Markdown-first article: minimal metadata + pre-rendered HTML content.
///
/// The `sys_title` field is the primary key — a URL-safe slug
/// used for lookups and routing (e.g., "linux-misc").
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub struct Article {
pub sys_title: String,
pub title: String,
pub published: bool,
pub author: String,
#[serde(default)]
pub tags: Vec<String>,
#[serde(default)]
pub published_on: Option<i64>,
/// Pre-rendered HTML from the .md file. Populated by ContentLoader.
#[serde(default)]
pub content_html: String,
}
// --- Legacy types for backwards compatibility with old JSON format ---
/// Legacy article format (nested JSON with sections/codes/etc.).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LegacyArticle {
pub title: String,
pub sys_title: String,
pub published: bool,
pub content: LegacyArticleContent,
#[serde(default)]
pub do_aws_sync: Option<bool>,
#[serde(default)]
pub published_on: Option<i64>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LegacyArticleContent {
pub author: String,
pub content: LegacyArticleBody,
#[serde(default, rename = "publishedOn")]
pub published_on: Option<i64>,
#[serde(default)]
pub references: Vec<LegacyReference>,
#[serde(default)]
pub tags: Vec<String>,
#[serde(default)]
pub title: Option<String>,
#[serde(default)]
pub systitle: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LegacyArticleBody {
#[serde(default)]
pub sections: Vec<LegacySection>,
#[serde(default)]
pub codes: Vec<serde_json::Value>,
#[serde(default)]
pub images: Vec<serde_json::Value>,
#[serde(default)]
pub img: Option<serde_json::Value>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LegacySection {
pub heading: String,
#[serde(default)]
pub paragraphs: Vec<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LegacyReference {
#[serde(default, rename = "dateAccessed")]
pub date_accessed: Option<i64>,
#[serde(default)]
pub title: Option<String>,
#[serde(default)]
pub url: Option<String>,
}
impl LegacyArticle {
/// Convert a legacy article to the new format by building HTML from sections.
pub fn to_article(&self) -> Article {
let mut html = String::new();
for section in &self.content.content.sections {
html.push_str(&format!("<h2>{}</h2>\n", section.heading));
for para in &section.paragraphs {
if !para.is_empty() {
html.push_str(&format!("<p>{}</p>\n", wo_md::markdown_to_html(para)));
}
}
}
Article {
sys_title: self.sys_title.clone(),
title: self.title.clone(),
published: self.published,
author: self.content.author.clone(),
tags: self.content.tags.clone(),
published_on: self.published_on.or(self.content.published_on),
content_html: html,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn deserialize_new_format() {
let json = r#"{
"sys_title": "test-article",
"title": "Test Article",
"published": true,
"author": "Author",
"tags": ["test"],
"published_on": 1000
}"#;
let article: Article = serde_json::from_str(json).unwrap();
assert_eq!(article.sys_title, "test-article");
assert_eq!(article.author, "Author");
assert_eq!(article.tags, vec!["test"]);
assert!(article.published);
assert_eq!(article.content_html, "");
}
#[test]
fn round_trip_bincode() {
let article = Article {
sys_title: "test".into(),
title: "Test".into(),
published: true,
author: "Author".into(),
tags: vec!["rust".into()],
published_on: Some(1000),
content_html: "<h1>Hello</h1>".into(),
};
let bytes = bincode::serialize(&article).unwrap();
let deserialized: Article = bincode::deserialize(&bytes).unwrap();
assert_eq!(article, deserialized);
}
#[test]
fn legacy_to_article() {
let json = r#"{
"title": "Legacy Article",
"sys_title": "legacy",
"published": true,
"content": {
"author": "Author",
"content": {
"sections": [
{"heading": "Intro", "paragraphs": ["Hello **world**."]}
],
"codes": [],
"images": []
},
"tags": ["test"],
"publishedOn": 1000
},
"published_on": 2000
}"#;
let legacy: LegacyArticle = serde_json::from_str(json).unwrap();
let article = legacy.to_article();
assert_eq!(article.sys_title, "legacy");
assert_eq!(article.author, "Author");
assert_eq!(article.tags, vec!["test"]);
assert_eq!(article.published_on, Some(2000));
assert!(article.content_html.contains("<h2>Intro</h2>"));
assert!(article.content_html.contains("<strong>world</strong>"));
}
}

5
reference/crates/wo-model/src/lib.rs
View file

@ -1,5 +0,0 @@
mod article;
mod loader;
pub use article::*;
pub use loader::ContentLoader;

249
reference/crates/wo-model/src/loader.rs
View file

@ -1,249 +0,0 @@
use std::fs;
use std::io;
use std::path::Path;
use crate::{Article, LegacyArticle};
/// Walks a content directory and loads all articles.
///
/// Supports two formats:
/// - **New**: minimal `.json` metadata + `.md` file for content
/// - **Legacy**: full JSON with nested sections/codes (falls back if no `.md`)
///
/// ```text
/// content_dir/
/// article-slug/
/// article-slug.json # metadata
/// article-slug.md # content (new format)
/// ```
pub struct ContentLoader;
impl ContentLoader {
/// Load all articles from the given content directory.
pub fn load_all(content_dir: &Path) -> io::Result<Vec<Article>> {
let mut articles = Vec::new();
let entries = fs::read_dir(content_dir)?;
for entry in entries {
let entry = entry?;
let path = entry.path();
if !path.is_dir() {
continue;
}
match Self::load_from_dir(&path) {
Ok(article) => articles.push(article),
Err(e) => {
eprintln!(
"wo-model: skipping {:?}: {}",
path.file_name().unwrap_or_default(),
e
);
}
}
}
articles.sort_by(|a, b| {
b.published_on
.unwrap_or(0)
.cmp(&a.published_on.unwrap_or(0))
});
Ok(articles)
}
/// Load a single article from a directory.
///
/// Tries new format first (minimal JSON + .md), falls back to legacy JSON.
pub fn load_from_dir(dir: &Path) -> io::Result<Article> {
let json_file = Self::find_json_file(dir)?;
let json_contents = fs::read_to_string(&json_file)?;
// Check if a .md file exists alongside the JSON.
let md_file = Self::find_md_file(dir);
if let Some(md_path) = md_file {
// New format: minimal JSON + markdown file.
// Try parsing as new format first, then fall back to legacy.
let mut article: Article =
if let Ok(a) = serde_json::from_str::<Article>(&json_contents) {
a
} else {
let legacy: LegacyArticle =
serde_json::from_str(&json_contents).map_err(|e| {
io::Error::new(
io::ErrorKind::InvalidData,
format!("{}: {}", json_file.display(), e),
)
})?;
legacy.to_article()
};
let md_content = fs::read_to_string(&md_path)?;
article.content_html = wo_md::markdown_to_html_block(&md_content);
Ok(article)
} else {
// Legacy format: full JSON with sections, no .md file.
let legacy: LegacyArticle = serde_json::from_str(&json_contents).map_err(|e| {
io::Error::new(
io::ErrorKind::InvalidData,
format!("{}: {}", json_file.display(), e),
)
})?;
Ok(legacy.to_article())
}
}
/// Load a single article from a specific JSON file path.
pub fn load_from_file(path: &Path) -> io::Result<Article> {
let dir = path.parent().unwrap_or(Path::new("."));
Self::load_from_dir(dir)
}
fn find_json_file(dir: &Path) -> io::Result<std::path::PathBuf> {
for entry in fs::read_dir(dir)? {
let entry = entry?;
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) == Some("json") {
return Ok(path);
}
}
Err(io::Error::new(
io::ErrorKind::NotFound,
format!("no .json file in {}", dir.display()),
))
}
fn find_md_file(dir: &Path) -> Option<std::path::PathBuf> {
let entries = fs::read_dir(dir).ok()?;
for entry in entries {
let entry = entry.ok()?;
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) == Some("md") {
return Some(path);
}
}
None
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
/// Create a new-format article: minimal JSON + .md file.
fn create_new_article(base: &Path, sys_title: &str, published: bool, tags: &[&str]) {
let dir = base.join(sys_title);
fs::create_dir_all(&dir).unwrap();
let tags_json: Vec<String> = tags.iter().map(|t| format!("\"{}\"", t)).collect();
let json = format!(
r#"{{
"sys_title": "{}",
"title": "{}",
"published": {},
"author": "Test Author",
"tags": [{}],
"published_on": 2000
}}"#,
sys_title, sys_title, published, tags_json.join(", ")
);
fs::write(dir.join(format!("{}.json", sys_title)), json).unwrap();
let md = format!("# {}\n\nThis is the **content** of the article.\n\n```rust\nfn main() {{}}\n```\n", sys_title);
fs::write(dir.join(format!("{}.md", sys_title)), md).unwrap();
}
/// Create a legacy-format article: full JSON, no .md file.
fn create_legacy_article(base: &Path, sys_title: &str, published: bool, tags: &[&str]) {
let dir = base.join(sys_title);
fs::create_dir_all(&dir).unwrap();
let tags_json: Vec<String> = tags.iter().map(|t| format!("\"{}\"", t)).collect();
let json = format!(
r#"{{
"title": "{}",
"sys_title": "{}",
"published": {},
"content": {{
"author": "Test Author",
"content": {{
"sections": [{{ "heading": "Intro", "paragraphs": ["Hello **world**."] }}],
"codes": [],
"images": []
}},
"tags": [{}],
"publishedOn": 1000
}},
"published_on": 2000
}}"#,
sys_title, sys_title, published, tags_json.join(", ")
);
fs::write(dir.join(format!("{}.json", sys_title)), json).unwrap();
}
#[test]
fn load_new_format() {
let tmp = tempfile::tempdir().unwrap();
create_new_article(tmp.path(), "my-article", true, &["rust"]);
let article = ContentLoader::load_from_dir(&tmp.path().join("my-article")).unwrap();
assert_eq!(article.sys_title, "my-article");
assert_eq!(article.author, "Test Author");
assert_eq!(article.tags, vec!["rust"]);
assert!(article.content_html.contains("<h1>my-article</h1>"));
assert!(article.content_html.contains("<strong>content</strong>"));
assert!(article.content_html.contains("<pre><code"));
}
#[test]
fn load_legacy_format() {
let tmp = tempfile::tempdir().unwrap();
create_legacy_article(tmp.path(), "legacy-art", true, &["test"]);
let article = ContentLoader::load_from_dir(&tmp.path().join("legacy-art")).unwrap();
assert_eq!(article.sys_title, "legacy-art");
assert_eq!(article.author, "Test Author");
assert!(article.content_html.contains("<h2>Intro</h2>"));
assert!(article.content_html.contains("<strong>world</strong>"));
}
#[test]
fn load_all_mixed() {
let tmp = tempfile::tempdir().unwrap();
create_new_article(tmp.path(), "new-art", true, &["rust"]);
create_legacy_article(tmp.path(), "old-art", true, &["go"]);
let articles = ContentLoader::load_all(tmp.path()).unwrap();
assert_eq!(articles.len(), 2);
let new = articles.iter().find(|a| a.sys_title == "new-art").unwrap();
assert!(new.content_html.contains("<h1>"));
let old = articles.iter().find(|a| a.sys_title == "old-art").unwrap();
assert!(old.content_html.contains("<h2>Intro</h2>"));
}
#[test]
fn skip_non_directory_entries() {
let tmp = tempfile::tempdir().unwrap();
create_new_article(tmp.path(), "valid-article", true, &[]);
fs::write(tmp.path().join("README.md"), "not an article").unwrap();
let articles = ContentLoader::load_all(tmp.path()).unwrap();
assert_eq!(articles.len(), 1);
}
#[test]
fn skip_directory_without_json() {
let tmp = tempfile::tempdir().unwrap();
create_new_article(tmp.path(), "valid-article", true, &[]);
fs::create_dir_all(tmp.path().join("images")).unwrap();
fs::write(tmp.path().join("images/photo.png"), b"fake image").unwrap();
let articles = ContentLoader::load_all(tmp.path()).unwrap();
assert_eq!(articles.len(), 1);
}
}

7
reference/crates/wo-route/Cargo.toml
View file

@ -1,7 +0,0 @@
[package]
name = "wo-route"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-http = { path = "../wo-http" }

5
reference/crates/wo-route/src/lib.rs
View file

@ -1,5 +0,0 @@
mod pattern;
mod router;
pub use pattern::{Pattern, Segment};
pub use router::{RouteParams, Router};

146
reference/crates/wo-route/src/pattern.rs
View file

@ -1,146 +0,0 @@
/// A compiled URL pattern segment.
#[derive(Debug, Clone, PartialEq)]
pub enum Segment {
/// Exact literal match (e.g., "blog").
Literal(String),
/// Named parameter (e.g., ":sys_title").
Param(String),
/// Wildcard matching the rest of the path (e.g., "*path").
Wildcard(String),
}
/// A compiled URL pattern like "/blog/:sys_title" or "/static/*path".
#[derive(Debug, Clone)]
pub struct Pattern {
pub segments: Vec<Segment>,
}
impl Pattern {
/// Compile a pattern string into segments.
///
/// - `/blog/:sys_title` → `[Literal("blog"), Param("sys_title")]`
/// - `/static/*path` → `[Literal("static"), Wildcard("path")]`
/// - `/` → `[]`
pub fn compile(pattern: &str) -> Self {
let segments = pattern
.trim_start_matches('/')
.split('/')
.filter(|s| !s.is_empty())
.map(|s| {
if let Some(name) = s.strip_prefix(':') {
Segment::Param(name.to_string())
} else if let Some(name) = s.strip_prefix('*') {
Segment::Wildcard(name.to_string())
} else {
Segment::Literal(s.to_string())
}
})
.collect();
Self { segments }
}
/// Try to match a URL path against this pattern.
///
/// Returns `Some(params)` if the path matches, where params is a list
/// of `(name, value)` pairs for any `:param` or `*wildcard` segments.
pub fn matches(&self, path: &str) -> Option<Vec<(String, String)>> {
let path_segments: Vec<&str> = path
.trim_start_matches('/')
.split('/')
.filter(|s| !s.is_empty())
.collect();
let mut params = Vec::new();
let mut pi = 0; // path segment index
for seg in &self.segments {
match seg {
Segment::Literal(lit) => {
if pi >= path_segments.len() || path_segments[pi] != lit.as_str() {
return None;
}
pi += 1;
}
Segment::Param(name) => {
if pi >= path_segments.len() {
return None;
}
params.push((name.clone(), path_segments[pi].to_string()));
pi += 1;
}
Segment::Wildcard(name) => {
if pi >= path_segments.len() {
return None;
}
let rest = path_segments[pi..].join("/");
params.push((name.clone(), rest));
return Some(params);
}
}
}
// All pattern segments consumed; path must also be fully consumed.
if pi == path_segments.len() {
Some(params)
} else {
None
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn root_path() {
let p = Pattern::compile("/");
assert!(p.matches("/").is_some());
assert!(p.matches("/blog").is_none());
}
#[test]
fn literal_path() {
let p = Pattern::compile("/about");
assert!(p.matches("/about").is_some());
assert!(p.matches("/contact").is_none());
assert!(p.matches("/about/extra").is_none());
}
#[test]
fn param_extraction() {
let p = Pattern::compile("/blog/:sys_title");
let params = p.matches("/blog/linux-misc").unwrap();
assert_eq!(params, vec![("sys_title".into(), "linux-misc".into())]);
assert!(p.matches("/blog").is_none());
assert!(p.matches("/blog/linux-misc/extra").is_none());
}
#[test]
fn wildcard() {
let p = Pattern::compile("/static/*path");
let params = p.matches("/static/styles/main.css").unwrap();
assert_eq!(params, vec![("path".into(), "styles/main.css".into())]);
let params = p.matches("/static/logo.png").unwrap();
assert_eq!(params, vec![("path".into(), "logo.png".into())]);
assert!(p.matches("/static").is_none());
}
#[test]
fn multi_segment() {
let p = Pattern::compile("/tag/:tag");
let params = p.matches("/tag/rust").unwrap();
assert_eq!(params, vec![("tag".into(), "rust".into())]);
}
#[test]
fn no_match() {
let p = Pattern::compile("/blog/:sys_title");
assert!(p.matches("/about").is_none());
assert!(p.matches("/").is_none());
}
}

127
reference/crates/wo-route/src/router.rs
View file

@ -1,127 +0,0 @@
use std::collections::HashMap;
use wo_http::request::{Method, Request};
use crate::pattern::Pattern;
/// Extracted route parameters.
#[derive(Debug, Clone, Default)]
pub struct RouteParams {
params: HashMap<String, String>,
}
impl RouteParams {
pub fn get(&self, key: &str) -> Option<&str> {
self.params.get(key).map(|s| s.as_str())
}
pub fn from_pairs(pairs: Vec<(String, String)>) -> Self {
Self {
params: pairs.into_iter().collect(),
}
}
}
/// A route entry: method + pattern + handler name.
struct Route {
method: Method,
pattern: Pattern,
handler: String,
}
/// URL router that matches requests to named handlers.
pub struct Router {
routes: Vec<Route>,
}
impl Router {
pub fn new() -> Self {
Self { routes: Vec::new() }
}
/// Add a route. Handler is a string name that the caller maps to a function.
pub fn add(&mut self, method: Method, pattern: &str, handler: &str) {
self.routes.push(Route {
method,
pattern: Pattern::compile(pattern),
handler: handler.to_string(),
});
}
/// Match a request to a route. Returns the handler name and extracted params.
pub fn dispatch(&self, request: &Request) -> Option<(String, RouteParams)> {
for route in &self.routes {
if route.method != request.method {
continue;
}
if let Some(pairs) = route.pattern.matches(&request.path) {
return Some((
route.handler.clone(),
RouteParams::from_pairs(pairs),
));
}
}
None
}
}
#[cfg(test)]
mod tests {
use super::*;
fn make_request(method: Method, path: &str) -> Request {
Request {
method,
path: path.to_string(),
query: None,
headers: HashMap::new(),
}
}
#[test]
fn dispatch_routes() {
let mut router = Router::new();
router.add(Method::Get, "/", "home");
router.add(Method::Get, "/blog/:sys_title", "article");
router.add(Method::Get, "/about", "about");
router.add(Method::Get, "/tag/:tag", "tag_listing");
router.add(Method::Get, "/static/*path", "static_file");
// Home.
let (handler, _) = router.dispatch(&make_request(Method::Get, "/")).unwrap();
assert_eq!(handler, "home");
// Article.
let (handler, params) = router
.dispatch(&make_request(Method::Get, "/blog/linux-misc"))
.unwrap();
assert_eq!(handler, "article");
assert_eq!(params.get("sys_title"), Some("linux-misc"));
// About.
let (handler, _) = router.dispatch(&make_request(Method::Get, "/about")).unwrap();
assert_eq!(handler, "about");
// Tag.
let (handler, params) = router
.dispatch(&make_request(Method::Get, "/tag/rust"))
.unwrap();
assert_eq!(handler, "tag_listing");
assert_eq!(params.get("tag"), Some("rust"));
// Static.
let (handler, params) = router
.dispatch(&make_request(Method::Get, "/static/styles/main.css"))
.unwrap();
assert_eq!(handler, "static_file");
assert_eq!(params.get("path"), Some("styles/main.css"));
// No match.
assert!(router
.dispatch(&make_request(Method::Get, "/nonexistent"))
.is_none());
// Wrong method.
assert!(router.dispatch(&make_request(Method::Post, "/")).is_none());
}
}

28
reference/crates/wo-rt/Cargo.toml
View file

@ -1,28 +0,0 @@
[package]
name = "wo-rt"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-model = { path = "../wo-model" }
wo-store = { path = "../wo-store" }
wo-watch = { path = "../wo-watch" }
wo-event = { path = "../wo-event" }
wo-sub = { path = "../wo-sub" }
wo-http = { path = "../wo-http" }
wo-route = { path = "../wo-route" }
wo-htmlx = { path = "../wo-htmlx" }
wo-md = { path = "../wo-md" }
wo-serve = { path = "../wo-serve" }
libc = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
bincode = { workspace = true }
[[bin]]
name = "writeonce"
path = "src/main.rs"
[dev-dependencies]
tempfile = "3"
bincode = { workspace = true }

31
reference/crates/wo-rt/src/config.rs
View file

@ -1,31 +0,0 @@
use std::path::PathBuf;
/// Runtime configuration.
#[derive(Debug, Clone)]
pub struct Config {
/// Path to the content directory (JSON + MD source files).
pub content_dir: PathBuf,
/// Path to the data directory (.seg + .idx derived files).
pub data_dir: PathBuf,
/// Path to the templates directory (.htmlx files).
pub templates_dir: PathBuf,
/// Path to the static assets directory (CSS, images).
pub static_dir: PathBuf,
/// HTTP bind address (e.g., "0.0.0.0:3000").
pub bind_addr: String,
/// Force a full rebuild on startup (ignore existing data/).
pub rebuild_on_start: bool,
}
impl Config {
pub fn new(content_dir: impl Into<PathBuf>, data_dir: impl Into<PathBuf>) -> Self {
Self {
content_dir: content_dir.into(),
data_dir: data_dir.into(),
templates_dir: PathBuf::from("templates"),
static_dir: PathBuf::from("static"),
bind_addr: "0.0.0.0:3000".into(),
rebuild_on_start: false,
}
}
}

130
reference/crates/wo-rt/src/handlers.rs
View file

@ -1,130 +0,0 @@
use wo_htmlx::{self, Value};
use wo_http::response::Response;
use wo_route::RouteParams;
use wo_store::Store;
use wo_htmlx::TemplateRegistry;
/// Render the homepage: list of published articles.
pub fn handle_home(store: &Store, templates: &TemplateRegistry) -> Response {
let articles = match store.list_published(0, 20) {
Ok(a) => a,
Err(e) => return Response::internal_error(&e.to_string()),
};
let articles_val: Vec<Value> = articles
.iter()
.map(|a| {
let json = serde_json::to_value(a).unwrap_or_default();
Value::from_json(&json)
})
.collect();
let mut ctx = std::collections::BTreeMap::new();
ctx.insert("articles".into(), Value::List(articles_val));
ctx.insert("page_title".into(), Value::String("writeonce".into()));
let context = Value::Object(ctx);
render_page("home", &context, templates)
}
/// Render a single article page.
pub fn handle_article(params: &RouteParams, store: &Store, templates: &TemplateRegistry) -> Response {
let sys_title = match params.get("sys_title") {
Some(t) => t,
None => return Response::not_found(),
};
let article = match store.get_by_title(sys_title) {
Ok(Some(a)) => a,
Ok(None) => return Response::not_found(),
Err(e) => return Response::internal_error(&e.to_string()),
};
let json = serde_json::to_value(&article).unwrap_or_default();
let mut ctx = std::collections::BTreeMap::new();
ctx.insert("article".into(), Value::from_json(&json));
ctx.insert("page_title".into(), Value::String(article.title.clone()));
let context = Value::Object(ctx);
render_page("article", &context, templates)
}
/// Render a tag listing page.
pub fn handle_tag(params: &RouteParams, store: &Store, templates: &TemplateRegistry) -> Response {
let tag = match params.get("tag") {
Some(t) => t,
None => return Response::not_found(),
};
let articles = match store.list_by_tag(tag) {
Ok(a) => a,
Err(e) => return Response::internal_error(&e.to_string()),
};
let articles_val: Vec<Value> = articles
.iter()
.map(|a| {
let json = serde_json::to_value(a).unwrap_or_default();
Value::from_json(&json)
})
.collect();
let mut ctx = std::collections::BTreeMap::new();
ctx.insert("articles".into(), Value::List(articles_val));
ctx.insert("page_title".into(), Value::String(format!("tag: {}", tag)));
ctx.insert("tag".into(), Value::String(tag.to_string()));
let context = Value::Object(ctx);
render_page("home", &context, templates)
}
/// Render a static page (about, contact).
pub fn handle_static_page(name: &str, templates: &TemplateRegistry) -> Response {
let mut ctx = std::collections::BTreeMap::new();
ctx.insert("page_title".into(), Value::String(name.to_string()));
let context = Value::Object(ctx);
render_page(name, &context, templates)
}
/// Render a page template composed with layout, header, and footer.
fn render_page(template_name: &str, context: &Value, templates: &TemplateRegistry) -> Response {
let page_nodes = match templates.get(template_name) {
Some(nodes) => nodes,
None => return Response::not_found(),
};
let partials = templates.partials();
// Render the page content.
let page_html = wo_htmlx::render(page_nodes, context, partials);
// Compose with layout if it exists.
let html = if let Some(layout_nodes) = templates.get("layout") {
// Inject page content and header/footer.
let header_html = templates
.get("header")
.map(|n| wo_htmlx::render(n, context, partials))
.unwrap_or_default();
let footer_html = templates
.get("footer")
.map(|n| wo_htmlx::render(n, context, partials))
.unwrap_or_default();
let mut layout_ctx = std::collections::BTreeMap::new();
layout_ctx.insert("content".into(), Value::String(page_html));
layout_ctx.insert("header".into(), Value::String(header_html));
layout_ctx.insert("footer".into(), Value::String(footer_html));
if let Value::Object(map) = context {
for (k, v) in map {
layout_ctx.insert(k.clone(), v.clone());
}
}
let layout_context = Value::Object(layout_ctx);
wo_htmlx::render(layout_nodes, &layout_context, partials)
} else {
page_html
};
Response::html(html)
}

6
reference/crates/wo-rt/src/lib.rs
View file

@ -1,6 +0,0 @@
mod config;
mod handlers;
mod runtime;
pub use config::Config;
pub use runtime::{Runtime, RuntimeHandle};

34
reference/crates/wo-rt/src/main.rs
View file

@ -1,34 +0,0 @@
use std::path::PathBuf;
use wo_rt::{Config, Runtime};
fn main() {
let content_dir = PathBuf::from("content");
let data_dir = PathBuf::from("data");
let config = Config {
content_dir,
data_dir,
templates_dir: PathBuf::from("templates"),
static_dir: PathBuf::from("static"),
bind_addr: "0.0.0.0:3000".into(),
rebuild_on_start: true,
};
let mut rt = match Runtime::new(&config) {
Ok(rt) => rt,
Err(e) => {
eprintln!("failed to start: {}", e);
std::process::exit(1);
}
};
eprintln!("writeonce started — content={} data={}",
config.content_dir.display(), config.data_dir.display());
if let Err(e) = rt.run() {
eprintln!("runtime error: {}", e);
std::process::exit(1);
}
eprintln!("writeonce shut down");
}

481
reference/crates/wo-rt/src/runtime.rs
View file

@ -1,481 +0,0 @@
use std::collections::HashMap;
use std::io;
use std::os::unix::io::RawFd;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
use wo_event::{EventLoop, Interest, SignalFd, TimerFd, Token};
use wo_htmlx::TemplateRegistry;
use wo_http::connection::{Connection, State};
use wo_http::listener::TcpListener;
use wo_http::request::Method;
use wo_http::response::Response;
use wo_route::{RouteParams, Router};
use wo_store::Store;
use wo_sub::{ChangeType, Subscription, SubscriptionManager};
use wo_watch::{ContentChange, ContentWatcher};
use crate::handlers;
use crate::Config;
// Well-known tokens.
const TOKEN_WATCHER: Token = 1;
const TOKEN_SIGNAL: Token = 2;
const TOKEN_TIMER: Token = 3;
const TOKEN_NOTIFY: Token = 4;
const TOKEN_HTTP_LISTENER: Token = 5;
// Token ranges.
const TOKEN_HTTP_BASE: Token = 10000;
const TOKEN_HTTP_MAX: Token = 19999;
const TOKEN_SUB_BASE: Token = 20000;
/// The writeonce runtime: single process, single event loop, all fds on one epoll.
pub struct Runtime {
event_loop: EventLoop,
store: Store,
watcher: ContentWatcher,
subscriptions: SubscriptionManager,
signal_fd: SignalFd,
timer_fd: TimerFd,
running: Arc<AtomicBool>,
// HTTP
listener: Option<TcpListener>,
connections: HashMap<Token, Connection>,
next_http_token: Token,
router: Router,
templates: TemplateRegistry,
static_dir: std::path::PathBuf,
// Subscribed browser connections.
next_sub_token: Token,
sub_token_to_fd: HashMap<Token, RawFd>,
}
/// Handle returned by `Runtime::start()` for external interaction.
pub struct RuntimeHandle {
running: Arc<AtomicBool>,
}
impl RuntimeHandle {
pub fn shutdown(&self) {
self.running.store(false, Ordering::SeqCst);
}
}
impl Runtime {
pub fn new(config: &Config) -> io::Result<Self> {
let event_loop = EventLoop::new()?;
let store = if config.rebuild_on_start {
let mut s = Store::open(&config.content_dir, &config.data_dir)?;
s.rebuild()?;
s
} else {
Store::open(&config.content_dir, &config.data_dir)?
};
let watcher = ContentWatcher::new(&config.content_dir)?;
let subscriptions = SubscriptionManager::new()?;
let signal_fd = SignalFd::new()?;
let timer_fd = TimerFd::new()?;
timer_fd.set(Duration::from_secs(60), Duration::from_secs(60))?;
// Register system fds.
event_loop.register(watcher.fd(), Interest::Readable, TOKEN_WATCHER)?;
event_loop.register(signal_fd.fd(), Interest::Readable, TOKEN_SIGNAL)?;
event_loop.register(timer_fd.fd(), Interest::Readable, TOKEN_TIMER)?;
event_loop.register(subscriptions.notify_fd(), Interest::Readable, TOKEN_NOTIFY)?;
// HTTP listener (optional — may not bind in test mode).
let listener = if !config.bind_addr.is_empty() {
match TcpListener::bind(&config.bind_addr) {
Ok(l) => {
event_loop.register(l.fd(), Interest::Readable, TOKEN_HTTP_LISTENER)?;
Some(l)
}
Err(e) => {
eprintln!("wo-rt: failed to bind {}: {}", config.bind_addr, e);
None
}
}
} else {
None
};
// Load templates.
let templates = TemplateRegistry::load(&config.templates_dir)?;
// Build router.
let mut router = Router::new();
router.add(Method::Get, "/", "home");
router.add(Method::Get, "/blog/:sys_title", "article");
router.add(Method::Get, "/about", "about");
router.add(Method::Get, "/contact", "contact");
router.add(Method::Get, "/tag/:tag", "tag_listing");
router.add(Method::Get, "/static/*path", "static_file");
Ok(Self {
event_loop,
store,
watcher,
subscriptions,
signal_fd,
timer_fd,
running: Arc::new(AtomicBool::new(false)),
listener,
connections: HashMap::new(),
next_http_token: TOKEN_HTTP_BASE,
router,
templates,
static_dir: config.static_dir.clone(),
next_sub_token: TOKEN_SUB_BASE,
sub_token_to_fd: HashMap::new(),
})
}
pub fn run(&mut self) -> io::Result<()> {
self.running.store(true, Ordering::SeqCst);
if let Some(ref listener) = self.listener {
let (_, port) = listener.local_addr()?;
eprintln!("writeonce listening on port {}", port);
}
while self.running.load(Ordering::SeqCst) {
let events = self.event_loop.poll(Some(Duration::from_millis(500)))?;
for event in events {
match event.token {
TOKEN_WATCHER if event.readable => {
self.handle_watcher_event()?;
}
TOKEN_SIGNAL if event.readable => {
let _ = self.signal_fd.read();
self.running.store(false, Ordering::SeqCst);
}
TOKEN_TIMER if event.readable => {
let _ = self.timer_fd.read();
}
TOKEN_NOTIFY if event.readable => {
// Notifications already written by SubscriptionManager.
}
TOKEN_HTTP_LISTENER if event.readable => {
self.handle_accept()?;
}
token if token >= TOKEN_SUB_BASE => {
// Subscribed connection.
if event.hangup || event.error {
if let Some(&fd) = self.sub_token_to_fd.get(&token) {
self.subscriptions.unsubscribe(fd);
let _ = self.event_loop.deregister(fd);
self.sub_token_to_fd.remove(&token);
// Don't close fd here — Connection owns it.
}
}
}
token if token >= TOKEN_HTTP_BASE => {
self.handle_http_event(token, &event)?;
}
_ => {}
}
}
// Clean up completed connections.
let done_tokens: Vec<Token> = self
.connections
.iter()
.filter(|(_, c)| c.state == State::Done)
.map(|(&t, _)| t)
.collect();
for token in done_tokens {
if let Some(conn) = self.connections.remove(&token) {
let _ = self.event_loop.deregister(conn.fd);
}
}
}
Ok(())
}
pub fn handle(&self) -> RuntimeHandle {
RuntimeHandle {
running: self.running.clone(),
}
}
pub fn store(&self) -> &Store {
&self.store
}
pub fn store_mut(&mut self) -> &mut Store {
&mut self.store
}
/// Subscribe a fd (binary format, for internal use).
pub fn subscribe(&mut self, fd: RawFd, query: Subscription) -> io::Result<Token> {
let token = self.next_sub_token;
self.next_sub_token += 1;
self.event_loop.register(fd, Interest::Readable, token)?;
self.subscriptions.subscribe(fd, query);
self.sub_token_to_fd.insert(token, fd);
Ok(token)
}
fn handle_accept(&mut self) -> io::Result<()> {
let listener = match &self.listener {
Some(l) => l,
None => return Ok(()),
};
// Accept all pending connections.
while let Some(client_fd) = listener.accept()? {
if self.next_http_token > TOKEN_HTTP_MAX {
self.next_http_token = TOKEN_HTTP_BASE; // recycle tokens
}
let token = self.next_http_token;
self.next_http_token += 1;
self.event_loop
.register(client_fd, Interest::Readable, token)?;
self.connections.insert(token, Connection::new(client_fd));
}
Ok(())
}
fn handle_http_event(&mut self, token: Token, event: &wo_event::Event) -> io::Result<()> {
if event.hangup || event.error {
if let Some(conn) = self.connections.remove(&token) {
let _ = self.event_loop.deregister(conn.fd);
}
return Ok(());
}
// Read phase: parse the request without holding mutable borrow during dispatch.
let conn = match self.connections.get_mut(&token) {
Some(c) => c,
None => return Ok(()),
};
match conn.state {
State::ReadingRequest if event.readable => {
let open = conn.read()?;
if !open {
conn.set_done();
return Ok(());
}
if let Some(request) = conn.try_parse() {
// Dispatch the handler (borrows self immutably via store/templates).
let dispatch_result = self.router.dispatch(&request);
let response = match dispatch_result {
Some((ref handler, ref params)) => {
dispatch_handler(handler, params, &self.store, &self.templates, &self.static_dir)
}
None => Response::not_found(),
};
// Now get the connection again to set the response.
let conn = self.connections.get_mut(&token).unwrap();
conn.set_response(&response);
if conn.write()? {
conn.set_done();
} else {
let fd = conn.fd;
self.event_loop.modify(fd, Interest::Writable, token)?;
}
}
}
State::WritingResponse if event.writable => {
if conn.write()? {
conn.set_done();
}
}
_ => {}
}
Ok(())
}
fn handle_watcher_event(&mut self) -> io::Result<()> {
let changes = self.watcher.process_events()?;
for change in &changes {
match change {
ContentChange::Created(sys_title) | ContentChange::Modified(sys_title) => {
self.store.rebuild()?;
let version = self.store.article_version(sys_title).unwrap_or(0);
let tags = self
.store
.get_by_title(sys_title)?
.map(|a| a.tags.clone())
.unwrap_or_default();
let change_type = match change {
ContentChange::Created(_) => ChangeType::Created,
_ => ChangeType::Updated,
};
self.subscriptions.notify(sys_title, &tags, change_type, version)?;
}
ContentChange::Deleted(sys_title) => {
let version = self.store.article_version(sys_title).unwrap_or(0);
self.subscriptions.notify(sys_title, &[], ChangeType::Deleted, version)?;
self.store.rebuild()?;
}
}
}
Ok(())
}
}
fn dispatch_handler(
handler: &str,
params: &RouteParams,
store: &Store,
templates: &TemplateRegistry,
static_dir: &std::path::Path,
) -> Response {
match handler {
"home" => handlers::handle_home(store, templates),
"article" => handlers::handle_article(params, store, templates),
"tag_listing" => handlers::handle_tag(params, store, templates),
"about" => handlers::handle_static_page("about", templates),
"contact" => handlers::handle_static_page("contact", templates),
"static_file" => {
let path = params.get("path").unwrap_or("");
wo_serve::send_file(static_dir, path)
}
_ => Response::not_found(),
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
fn create_test_article(base: &std::path::Path, sys_title: &str, published: bool) {
let dir = base.join(sys_title);
fs::create_dir_all(&dir).unwrap();
let json = format!(
r#"{{
"title": "{}",
"sys_title": "{}",
"published": {},
"content": {{
"author": "Author",
"content": {{
"sections": [{{ "heading": "Intro", "paragraphs": ["Hello."] }}],
"codes": [],
"images": []
}},
"tags": ["test"],
"publishedOn": 1000
}},
"published_on": 1000
}}"#,
sys_title, sys_title, published
);
fs::write(dir.join(format!("{}.json", sys_title)), json).unwrap();
}
fn test_config(content: &std::path::Path, data: &std::path::Path) -> Config {
Config {
bind_addr: String::new(), // no HTTP in tests
..Config::new(content, data)
}
}
#[test]
fn runtime_creates_and_queries() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let data = tmp.path().join("data");
create_test_article(&content, "art-one", true);
create_test_article(&content, "art-two", true);
let config = test_config(&content, &data);
let rt = Runtime::new(&config).unwrap();
let art = rt.store().get_by_title("art-one").unwrap().unwrap();
assert_eq!(art.sys_title, "art-one");
assert_eq!(rt.store().count_published().unwrap(), 2);
}
#[test]
fn runtime_subscribe_and_notify() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let data = tmp.path().join("data");
create_test_article(&content, "art-one", true);
let config = test_config(&content, &data);
let mut rt = Runtime::new(&config).unwrap();
let mut fds = [0i32; 2];
assert_eq!(unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_NONBLOCK) }, 0);
let (read_fd, write_fd) = (fds[0], fds[1]);
rt.subscribe(write_fd, Subscription::ByTitle("art-one".into())).unwrap();
rt.subscriptions
.notify("art-one", &["test".into()], ChangeType::Updated, 1)
.unwrap();
let mut len_buf = [0u8; 4];
let ret = unsafe { libc::read(read_fd, len_buf.as_mut_ptr() as *mut libc::c_void, 4) };
assert_eq!(ret, 4);
let len = u32::from_le_bytes(len_buf) as usize;
let mut payload = vec![0u8; len];
let ret = unsafe { libc::read(read_fd, payload.as_mut_ptr() as *mut libc::c_void, len) };
assert_eq!(ret as usize, len);
let notification: wo_sub::Notification = bincode::deserialize(&payload).unwrap();
assert_eq!(notification.sys_title, "art-one");
unsafe { libc::close(read_fd); libc::close(write_fd); }
}
#[test]
fn runtime_shutdown_via_handle() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let data = tmp.path().join("data");
fs::create_dir_all(&content).unwrap();
let config = test_config(&content, &data);
let mut rt = Runtime::new(&config).unwrap();
let handle = rt.handle();
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(100));
handle.shutdown();
});
rt.run().unwrap();
}
#[test]
fn runtime_rebuild_on_start() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let data = tmp.path().join("data");
create_test_article(&content, "art-one", true);
let config = test_config(&content, &data);
let rt = Runtime::new(&config).unwrap();
assert_eq!(rt.store().count_published().unwrap(), 1);
drop(rt);
let config = Config {
rebuild_on_start: true,
..test_config(&content, &data)
};
let rt = Runtime::new(&config).unwrap();
assert_eq!(rt.store().count_published().unwrap(), 1);
}
}

13
reference/crates/wo-seg/Cargo.toml
View file

@ -1,13 +0,0 @@
[package]
name = "wo-seg"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-model = { path = "../wo-model" }
serde = { workspace = true }
bincode = { workspace = true }
libc = { workspace = true }
[dev-dependencies]
tempfile = "3"

111
reference/crates/wo-seg/src/header.rs
View file

@ -1,111 +0,0 @@
use std::io;
/// Magic bytes: "WOSF" (WriteOnce Segment File).
pub const MAGIC: [u8; 4] = *b"WOSF";
/// Segment file version.
pub const VERSION: u16 = 1;
/// Fixed header size in bytes.
pub const HEADER_SIZE: u64 = 32;
/// The segment file header, stored at byte 0 of every .seg file.
///
/// Layout (32 bytes):
/// ```text
/// [0..4] magic: b"WOSF"
/// [4..6] version: u16 LE
/// [6..8] flags: u16 LE (reserved)
/// [8..16] record_count: u64 LE
/// [16..24] data_start: u64 LE
/// [24..32] reserved: 8 bytes
/// ```
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct SegHeader {
pub version: u16,
pub flags: u16,
pub record_count: u64,
pub data_start: u64,
}
impl SegHeader {
/// Create a new header with default values.
pub fn new() -> Self {
Self {
version: VERSION,
flags: 0,
record_count: 0,
data_start: HEADER_SIZE,
}
}
/// Serialize the header into a 32-byte buffer.
pub fn to_bytes(&self) -> [u8; 32] {
let mut buf = [0u8; 32];
buf[0..4].copy_from_slice(&MAGIC);
buf[4..6].copy_from_slice(&self.version.to_le_bytes());
buf[6..8].copy_from_slice(&self.flags.to_le_bytes());
buf[8..16].copy_from_slice(&self.record_count.to_le_bytes());
buf[16..24].copy_from_slice(&self.data_start.to_le_bytes());
// [24..32] reserved, stays zero
buf
}
/// Parse a header from a 32-byte buffer.
pub fn from_bytes(buf: &[u8; 32]) -> io::Result<Self> {
if &buf[0..4] != &MAGIC {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!(
"invalid magic bytes: expected {:?}, got {:?}",
MAGIC,
&buf[0..4]
),
));
}
let version = u16::from_le_bytes([buf[4], buf[5]]);
if version != VERSION {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("unsupported version: {}", version),
));
}
let flags = u16::from_le_bytes([buf[6], buf[7]]);
let record_count = u64::from_le_bytes(buf[8..16].try_into().unwrap());
let data_start = u64::from_le_bytes(buf[16..24].try_into().unwrap());
Ok(Self {
version,
flags,
record_count,
data_start,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn round_trip() {
let header = SegHeader {
version: VERSION,
flags: 0,
record_count: 42,
data_start: HEADER_SIZE,
};
let bytes = header.to_bytes();
let parsed = SegHeader::from_bytes(&bytes).unwrap();
assert_eq!(header, parsed);
}
#[test]
fn bad_magic() {
let mut bytes = SegHeader::new().to_bytes();
bytes[0] = b'X';
assert!(SegHeader::from_bytes(&bytes).is_err());
}
}

14
reference/crates/wo-seg/src/lib.rs
View file

@ -1,14 +0,0 @@
mod header;
mod reader;
mod writer;
pub use header::{SegHeader, HEADER_SIZE, MAGIC};
pub use reader::SegReader;
pub use writer::SegWriter;
/// Record flags.
pub const FLAG_ACTIVE: u8 = 0x00;
pub const FLAG_TOMBSTONED: u8 = 0x01;
/// Size of the per-record header: 4 bytes length + 1 byte flags.
pub const RECORD_HEADER_SIZE: usize = 5;

221
reference/crates/wo-seg/src/reader.rs
View file

@ -1,221 +0,0 @@
use std::fs::File;
use std::io;
use std::os::unix::io::AsRawFd;
use std::path::Path;
use wo_model::Article;
use crate::header::SegHeader;
use crate::{FLAG_TOMBSTONED, RECORD_HEADER_SIZE};
/// Reads articles from a .seg file.
pub struct SegReader {
file: File,
header: SegHeader,
}
impl SegReader {
/// Open an existing .seg file and validate its header.
pub fn open(path: &Path) -> io::Result<Self> {
let file = File::open(path)?;
let mut header_buf = [0u8; 32];
pread_exact(&file, &mut header_buf, 0)?;
let header = SegHeader::from_bytes(&header_buf)?;
Ok(Self { file, header })
}
/// Read a single article record at the given byte offset.
///
/// Returns `None` if the record is tombstoned.
pub fn read_at(&self, offset: u64) -> io::Result<Option<Article>> {
// Read the record header: [u32 length][u8 flags]
let mut rec_header = [0u8; RECORD_HEADER_SIZE];
pread_exact(&self.file, &mut rec_header, offset)?;
let payload_len = u32::from_le_bytes(rec_header[0..4].try_into().unwrap()) as usize;
let flags = rec_header[4];
if flags == FLAG_TOMBSTONED {
return Ok(None);
}
// Read the payload.
let mut payload = vec![0u8; payload_len];
pread_exact(&self.file, &mut payload, offset + RECORD_HEADER_SIZE as u64)?;
let article: Article = bincode::deserialize(&payload).map_err(|e| {
io::Error::new(io::ErrorKind::InvalidData, format!("bincode: {}", e))
})?;
Ok(Some(article))
}
/// Iterate all active (non-tombstoned) records.
///
/// Returns `(byte_offset, Article)` pairs.
pub fn iter(&self) -> io::Result<Vec<(u64, Article)>> {
let mut results = Vec::new();
let mut pos = self.header.data_start;
for _ in 0..self.header.record_count {
// Read record header.
let mut rec_header = [0u8; RECORD_HEADER_SIZE];
pread_exact(&self.file, &mut rec_header, pos)?;
let payload_len = u32::from_le_bytes(rec_header[0..4].try_into().unwrap()) as usize;
let flags = rec_header[4];
let record_offset = pos;
pos += RECORD_HEADER_SIZE as u64 + payload_len as u64;
if flags == FLAG_TOMBSTONED {
continue;
}
let mut payload = vec![0u8; payload_len];
pread_exact(
&self.file,
&mut payload,
record_offset + RECORD_HEADER_SIZE as u64,
)?;
let article: Article = bincode::deserialize(&payload).map_err(|e| {
io::Error::new(io::ErrorKind::InvalidData, format!("bincode: {}", e))
})?;
results.push((record_offset, article));
}
Ok(results)
}
/// Return the parsed file header.
pub fn header(&self) -> &SegHeader {
&self.header
}
}
/// Positional read using pread(2). Does not modify the file offset.
fn pread_exact(file: &File, buf: &mut [u8], offset: u64) -> io::Result<()> {
let fd = file.as_raw_fd();
let mut read = 0usize;
while read < buf.len() {
let ret = unsafe {
libc::pread(
fd,
buf[read..].as_mut_ptr() as *mut libc::c_void,
buf.len() - read,
(offset + read as u64) as libc::off_t,
)
};
if ret < 0 {
return Err(io::Error::last_os_error());
}
if ret == 0 {
return Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
"pread returned 0",
));
}
read += ret as usize;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::SegWriter;
use wo_model::*;
fn sample_article(sys_title: &str) -> Article {
Article {
sys_title: sys_title.to_string(),
title: format!("Title: {}", sys_title),
published: true,
author: "Author".into(),
tags: vec!["test".into()],
published_on: Some(2000),
content_html: "<h1>Hello world.</h1>".into(),
}
}
#[test]
fn write_and_read_single() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("test.seg");
let article = sample_article("test-one");
let offset = {
let mut writer = SegWriter::create(&path).unwrap();
writer.append(&article).unwrap()
};
let reader = SegReader::open(&path).unwrap();
assert_eq!(reader.header().record_count, 1);
let loaded = reader.read_at(offset).unwrap().unwrap();
assert_eq!(loaded.sys_title, "test-one");
assert_eq!(loaded, article);
}
#[test]
fn write_and_read_multiple() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("multi.seg");
let articles: Vec<Article> = (0..5).map(|i| sample_article(&format!("art-{}", i))).collect();
let offsets: Vec<u64> = {
let mut writer = SegWriter::create(&path).unwrap();
articles.iter().map(|a| writer.append(a).unwrap()).collect()
};
let reader = SegReader::open(&path).unwrap();
assert_eq!(reader.header().record_count, 5);
// Read each by offset.
for (i, offset) in offsets.iter().enumerate() {
let loaded = reader.read_at(*offset).unwrap().unwrap();
assert_eq!(loaded.sys_title, format!("art-{}", i));
}
// Iterate all.
let all = reader.iter().unwrap();
assert_eq!(all.len(), 5);
}
#[test]
fn tombstone_record() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("tomb.seg");
let offset = {
let mut writer = SegWriter::create(&path).unwrap();
let o = writer.append(&sample_article("to-delete")).unwrap();
writer.append(&sample_article("to-keep")).unwrap();
writer.tombstone(o).unwrap();
o
};
let reader = SegReader::open(&path).unwrap();
// Direct read returns None for tombstoned.
assert!(reader.read_at(offset).unwrap().is_none());
// Iter skips tombstoned.
let all = reader.iter().unwrap();
assert_eq!(all.len(), 1);
assert_eq!(all[0].1.sys_title, "to-keep");
}
#[test]
fn corrupted_header() {
let tmp = tempfile::tempdir().unwrap();
let path = tmp.path().join("bad.seg");
std::fs::write(&path, b"not a segment file at all!!!!!xx").unwrap();
assert!(SegReader::open(&path).is_err());
}
}

129
reference/crates/wo-seg/src/writer.rs
View file

@ -1,129 +0,0 @@
use std::fs::{File, OpenOptions};
use std::io;
use std::os::unix::io::AsRawFd;
use std::path::Path;
use wo_model::Article;
use crate::header::{SegHeader, HEADER_SIZE};
use crate::{FLAG_ACTIVE, RECORD_HEADER_SIZE};
/// Writes articles to a .seg file.
///
/// Records are appended sequentially. Each append returns the byte offset
/// of the record, which can be stored in an index for direct access.
pub struct SegWriter {
file: File,
header: SegHeader,
/// Current write position (end of file).
pos: u64,
}
impl SegWriter {
/// Create a new .seg file at the given path.
///
/// Writes the initial header and optionally pre-allocates disk space
/// using `fallocate` to reduce fragmentation.
pub fn create(path: &Path) -> io::Result<Self> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(path)?;
let header = SegHeader::new();
// Write the header at position 0.
pwrite_all(&file, &header.to_bytes(), 0)?;
// Pre-allocate 1 MB to reduce fragmentation.
let _ = fallocate_safe(&file, HEADER_SIZE as i64, 1024 * 1024);
Ok(Self {
file,
header,
pos: HEADER_SIZE,
})
}
/// Append an article as an active record.
///
/// Returns the byte offset of the record start (the position of the
/// length prefix), which can be used for direct reads via `SegReader::read_at`.
pub fn append(&mut self, article: &Article) -> io::Result<u64> {
let payload = bincode::serialize(article).map_err(|e| {
io::Error::new(io::ErrorKind::InvalidData, format!("bincode: {}", e))
})?;
let record_offset = self.pos;
let payload_len = payload.len() as u32;
// Build record: [u32 length][u8 flags][payload]
let mut record = Vec::with_capacity(RECORD_HEADER_SIZE + payload.len());
record.extend_from_slice(&payload_len.to_le_bytes());
record.push(FLAG_ACTIVE);
record.extend_from_slice(&payload);
pwrite_all(&self.file, &record, record_offset)?;
self.pos += record.len() as u64;
self.header.record_count += 1;
// Update the header with the new record count.
pwrite_all(&self.file, &self.header.to_bytes(), 0)?;
Ok(record_offset)
}
/// Tombstone a record at the given offset.
///
/// Sets the flags byte to `FLAG_TOMBSTONED` without modifying the payload.
pub fn tombstone(&self, record_offset: u64) -> io::Result<()> {
let flags_offset = record_offset + 4; // skip the u32 length
pwrite_all(&self.file, &[crate::FLAG_TOMBSTONED], flags_offset)
}
/// Return the current header (record count, etc.).
pub fn header(&self) -> &SegHeader {
&self.header
}
/// Sync file data and metadata to disk.
pub fn sync(&self) -> io::Result<()> {
self.file.sync_all()
}
}
/// Positional write using pwrite(2). Does not modify the file offset.
fn pwrite_all(file: &File, buf: &[u8], offset: u64) -> io::Result<()> {
let fd = file.as_raw_fd();
let mut written = 0usize;
while written < buf.len() {
let ret = unsafe {
libc::pwrite(
fd,
buf[written..].as_ptr() as *const libc::c_void,
buf.len() - written,
(offset + written as u64) as libc::off_t,
)
};
if ret < 0 {
return Err(io::Error::last_os_error());
}
if ret == 0 {
return Err(io::Error::new(io::ErrorKind::WriteZero, "pwrite returned 0"));
}
written += ret as usize;
}
Ok(())
}
/// Best-effort fallocate. Non-fatal if unsupported.
fn fallocate_safe(file: &File, offset: i64, len: i64) -> io::Result<()> {
let ret = unsafe { libc::fallocate(file.as_raw_fd(), 0, offset, len) };
if ret < 0 {
Err(io::Error::last_os_error())
} else {
Ok(())
}
}

11
reference/crates/wo-serve/Cargo.toml
View file

@ -1,11 +0,0 @@
[package]
name = "wo-serve"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-http = { path = "../wo-http" }
libc = { workspace = true }
[dev-dependencies]
tempfile = "3"

7
reference/crates/wo-serve/src/lib.rs
View file

@ -1,7 +0,0 @@
mod mime;
mod resolve;
mod sendfile;
pub use mime::content_type_for;
pub use resolve::resolve_path;
pub use sendfile::send_file;

44
reference/crates/wo-serve/src/mime.rs
View file

@ -1,44 +0,0 @@
/// Map a file extension to its HTTP Content-Type.
pub fn content_type_for(path: &str) -> &'static str {
let ext = path.rsplit('.').next().unwrap_or("");
match ext {
"html" | "htmlx" => "text/html; charset=utf-8",
"css" => "text/css; charset=utf-8",
"js" => "application/javascript; charset=utf-8",
"json" => "application/json; charset=utf-8",
"png" => "image/png",
"jpg" | "jpeg" => "image/jpeg",
"gif" => "image/gif",
"svg" => "image/svg+xml",
"ico" => "image/x-icon",
"woff" => "font/woff",
"woff2" => "font/woff2",
"ttf" => "font/ttf",
"txt" => "text/plain; charset=utf-8",
"xml" => "application/xml; charset=utf-8",
_ => "application/octet-stream",
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn known_types() {
assert_eq!(content_type_for("main.css"), "text/css; charset=utf-8");
assert_eq!(content_type_for("logo.png"), "image/png");
assert_eq!(content_type_for("app.js"), "application/javascript; charset=utf-8");
assert_eq!(content_type_for("favicon.ico"), "image/x-icon");
}
#[test]
fn unknown_type() {
assert_eq!(content_type_for("file.xyz"), "application/octet-stream");
}
#[test]
fn nested_path() {
assert_eq!(content_type_for("styles/code-theme.css"), "text/css; charset=utf-8");
}
}

80
reference/crates/wo-serve/src/resolve.rs
View file

@ -1,80 +0,0 @@
use std::io;
use std::path::{Path, PathBuf};
/// Resolve a request path relative to a static directory.
///
/// Returns the absolute path to the file, or an error if:
/// - The path contains `..` (directory traversal)
/// - The resolved path escapes the base directory
/// - The file doesn't exist
pub fn resolve_path(base: &Path, request_path: &str) -> io::Result<PathBuf> {
// Reject any path with `..`
if request_path.contains("..") {
return Err(io::Error::new(
io::ErrorKind::PermissionDenied,
"path traversal rejected",
));
}
let clean = request_path.trim_start_matches('/');
let resolved = base.join(clean);
// Canonicalize and verify it's still under base.
let canonical = resolved.canonicalize().map_err(|_| {
io::Error::new(io::ErrorKind::NotFound, "file not found")
})?;
let canonical_base = base.canonicalize().map_err(|_| {
io::Error::new(io::ErrorKind::NotFound, "base directory not found")
})?;
if !canonical.starts_with(&canonical_base) {
return Err(io::Error::new(
io::ErrorKind::PermissionDenied,
"path traversal rejected",
));
}
if !canonical.is_file() {
return Err(io::Error::new(io::ErrorKind::NotFound, "not a file"));
}
Ok(canonical)
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
#[test]
fn valid_path() {
let tmp = tempfile::tempdir().unwrap();
fs::write(tmp.path().join("test.css"), "body {}").unwrap();
let path = resolve_path(tmp.path(), "test.css").unwrap();
assert!(path.ends_with("test.css"));
}
#[test]
fn nested_path() {
let tmp = tempfile::tempdir().unwrap();
fs::create_dir_all(tmp.path().join("styles")).unwrap();
fs::write(tmp.path().join("styles/main.css"), "body {}").unwrap();
let path = resolve_path(tmp.path(), "styles/main.css").unwrap();
assert!(path.ends_with("styles/main.css"));
}
#[test]
fn reject_traversal() {
let tmp = tempfile::tempdir().unwrap();
assert!(resolve_path(tmp.path(), "../etc/passwd").is_err());
}
#[test]
fn reject_nonexistent() {
let tmp = tempfile::tempdir().unwrap();
assert!(resolve_path(tmp.path(), "nonexistent.css").is_err());
}
}

109
reference/crates/wo-serve/src/sendfile.rs
View file

@ -1,109 +0,0 @@
use std::fs;
use std::io;
use std::os::unix::io::RawFd;
use std::path::Path;
use wo_http::response::Response;
use crate::mime::content_type_for;
use crate::resolve::resolve_path;
/// Serve a static file. Returns a Response with the file contents.
///
/// For small files, reads into memory and returns a normal Response.
/// For zero-copy serving via sendfile, use `send_file_zero_copy` instead.
pub fn send_file(static_dir: &Path, request_path: &str) -> Response {
let file_path = match resolve_path(static_dir, request_path) {
Ok(p) => p,
Err(e) => {
return match e.kind() {
io::ErrorKind::NotFound => Response::not_found(),
io::ErrorKind::PermissionDenied => {
Response::new(403, "Forbidden")
}
_ => Response::internal_error(&e.to_string()),
};
}
};
let body = match fs::read(&file_path) {
Ok(b) => b,
Err(_) => return Response::internal_error("failed to read file"),
};
let ct = content_type_for(file_path.to_str().unwrap_or(""));
let mut response = Response::ok(body, ct);
response.header("Cache-Control", "public, max-age=3600");
response
}
/// Zero-copy file serving using sendfile(2).
///
/// Sends the file directly from the page cache to the socket fd.
/// Returns the number of bytes sent, or an error.
pub fn sendfile_to_fd(socket_fd: RawFd, file_path: &Path) -> io::Result<usize> {
let file = fs::File::open(file_path)?;
let file_fd = {
use std::os::unix::io::AsRawFd;
file.as_raw_fd()
};
let metadata = file.metadata()?;
let file_size = metadata.len() as usize;
let mut offset: libc::off_t = 0;
let mut sent = 0usize;
while sent < file_size {
let n = unsafe {
libc::sendfile(
socket_fd,
file_fd,
&mut offset,
file_size - sent,
)
};
if n < 0 {
let err = io::Error::last_os_error();
if err.raw_os_error() == Some(libc::EAGAIN) {
break; // partial send, caller should retry on EPOLLOUT
}
return Err(err);
}
if n == 0 {
break;
}
sent += n as usize;
}
Ok(sent)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn serve_existing_file() {
let tmp = tempfile::tempdir().unwrap();
fs::write(tmp.path().join("test.css"), "body { color: red; }").unwrap();
let response = send_file(tmp.path(), "test.css");
assert_eq!(response.status, 200);
assert!(String::from_utf8_lossy(&response.body).contains("color: red"));
}
#[test]
fn serve_nonexistent_file() {
let tmp = tempfile::tempdir().unwrap();
let response = send_file(tmp.path(), "nope.css");
assert_eq!(response.status, 404);
}
#[test]
fn serve_traversal_attempt() {
let tmp = tempfile::tempdir().unwrap();
let response = send_file(tmp.path(), "../etc/passwd");
assert_eq!(response.status, 403);
}
}

13
reference/crates/wo-store/Cargo.toml
View file

@ -1,13 +0,0 @@
[package]
name = "wo-store"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-model = { path = "../wo-model" }
wo-seg = { path = "../wo-seg" }
wo-index = { path = "../wo-index" }
serde_json = { workspace = true }
[dev-dependencies]
tempfile = "3"

355
reference/crates/wo-store/src/lib.rs
View file

@ -1,355 +0,0 @@
use std::collections::HashMap;
use std::io;
use std::path::{Path, PathBuf};
use wo_index::{DateIndex, TagIndex, TitleIndex};
use wo_model::{Article, ContentLoader};
use wo_seg::{SegReader, SegWriter};
/// Unified storage engine composing .seg files and indexes.
///
/// Provides the query API that the rest of the system uses.
/// Handles cold-start (rebuild from content/) and incremental ingestion.
pub struct Store {
seg_path: PathBuf,
index_dir: PathBuf,
content_dir: PathBuf,
title_idx: TitleIndex,
date_idx: DateIndex,
tag_idx: TagIndex,
/// Monotonic version counter per sys_title (for subscription diffs).
versions: HashMap<String, u64>,
version_counter: u64,
}
impl Store {
/// Open an existing data directory, or cold-start from content/.
///
/// If `data_dir` contains valid .seg and index files, opens them.
/// Otherwise, rebuilds everything from `content_dir`.
pub fn open(content_dir: &Path, data_dir: &Path) -> io::Result<Self> {
let seg_path = data_dir.join("articles.seg");
let index_dir = data_dir.join("index");
if seg_path.exists() && index_dir.join("title.idx").exists() {
Self::open_existing(content_dir, &seg_path, &index_dir)
} else {
let mut store = Self::init_empty(content_dir, &seg_path, &index_dir)?;
store.rebuild()?;
Ok(store)
}
}
fn open_existing(
content_dir: &Path,
seg_path: &Path,
index_dir: &Path,
) -> io::Result<Self> {
let title_idx = TitleIndex::open(&index_dir.join("title.idx"))?;
let date_idx = DateIndex::open(&index_dir.join("date.idx"))?;
let tag_idx = TagIndex::open(&index_dir.join("tags.idx"))?;
// Build version map from existing seg records.
let reader = SegReader::open(seg_path)?;
let mut versions = HashMap::new();
let mut version_counter = 0u64;
for (_, article) in reader.iter()? {
version_counter += 1;
versions.insert(article.sys_title.clone(), version_counter);
}
Ok(Self {
seg_path: seg_path.to_path_buf(),
index_dir: index_dir.to_path_buf(),
content_dir: content_dir.to_path_buf(),
title_idx,
date_idx,
tag_idx,
versions,
version_counter,
})
}
fn init_empty(
content_dir: &Path,
seg_path: &Path,
index_dir: &Path,
) -> io::Result<Self> {
std::fs::create_dir_all(seg_path.parent().unwrap())?;
std::fs::create_dir_all(index_dir)?;
// Create empty seg file.
let _writer = SegWriter::create(seg_path)?;
// Create empty indexes.
TitleIndex::build(&index_dir.join("title.idx"), &[])?;
DateIndex::build(&index_dir.join("date.idx"), &mut [])?;
TagIndex::build(&index_dir.join("tags.idx"), &[])?;
let title_idx = TitleIndex::open(&index_dir.join("title.idx"))?;
let date_idx = DateIndex::open(&index_dir.join("date.idx"))?;
let tag_idx = TagIndex::open(&index_dir.join("tags.idx"))?;
Ok(Self {
seg_path: seg_path.to_path_buf(),
index_dir: index_dir.to_path_buf(),
content_dir: content_dir.to_path_buf(),
title_idx,
date_idx,
tag_idx,
versions: HashMap::new(),
version_counter: 0,
})
}
/// Full rebuild: load all articles from content/, rewrite .seg and indexes.
pub fn rebuild(&mut self) -> io::Result<()> {
let articles = ContentLoader::load_all(&self.content_dir)?;
// Write all articles to a new .seg file.
let mut writer = SegWriter::create(&self.seg_path)?;
let mut index_entries: Vec<(String, i64, Vec<String>, u64)> = Vec::new();
self.versions.clear();
self.version_counter = 0;
for article in &articles {
let offset = writer.append(article)?;
let timestamp = article.published_on.unwrap_or(0);
index_entries.push((
article.sys_title.clone(),
timestamp,
article.tags.clone(),
offset,
));
self.version_counter += 1;
self.versions
.insert(article.sys_title.clone(), self.version_counter);
}
writer.sync()?;
// Rebuild all indexes.
wo_index::rebuild_all(&self.index_dir, &index_entries)?;
// Reload indexes.
self.title_idx = TitleIndex::open(&self.index_dir.join("title.idx"))?;
self.date_idx = DateIndex::open(&self.index_dir.join("date.idx"))?;
self.tag_idx = TagIndex::open(&self.index_dir.join("tags.idx"))?;
Ok(())
}
/// Look up a single article by sys_title.
pub fn get_by_title(&self, sys_title: &str) -> io::Result<Option<Article>> {
match self.title_idx.get(sys_title) {
Some(offset) => {
let reader = SegReader::open(&self.seg_path)?;
reader.read_at(offset)
}
None => Ok(None),
}
}
/// List published articles, most recent first, with pagination.
pub fn list_published(&self, skip: usize, limit: usize) -> io::Result<Vec<Article>> {
let offsets = self.date_idx.latest(skip + limit);
let reader = SegReader::open(&self.seg_path)?;
let mut articles = Vec::new();
for &offset in offsets.iter().skip(skip).take(limit) {
if let Some(article) = reader.read_at(offset)? {
if article.published {
articles.push(article);
}
}
}
Ok(articles)
}
/// List all articles with a given tag.
pub fn list_by_tag(&self, tag: &str) -> io::Result<Vec<Article>> {
let offsets = match self.tag_idx.get(tag) {
Some(offsets) => offsets,
None => return Ok(vec![]),
};
let reader = SegReader::open(&self.seg_path)?;
let mut articles = Vec::new();
for &offset in offsets {
if let Some(article) = reader.read_at(offset)? {
articles.push(article);
}
}
Ok(articles)
}
/// List articles published within a timestamp range.
pub fn list_by_date_range(&self, start: i64, end: i64) -> io::Result<Vec<Article>> {
let offsets = self.date_idx.range(start, end);
let reader = SegReader::open(&self.seg_path)?;
let mut articles = Vec::new();
for offset in offsets {
if let Some(article) = reader.read_at(offset)? {
articles.push(article);
}
}
Ok(articles)
}
/// Count of published articles.
pub fn count_published(&self) -> io::Result<usize> {
// For now, iterate and count. With small article counts this is fine.
let reader = SegReader::open(&self.seg_path)?;
let count = reader
.iter()?
.into_iter()
.filter(|(_, a)| a.published)
.count();
Ok(count)
}
/// Ingest a single article from a JSON file path.
///
/// Appends to .seg and triggers a full index rebuild.
/// Returns the sys_title of the ingested article.
pub fn ingest_article(&mut self, json_path: &Path) -> io::Result<String> {
let article = ContentLoader::load_from_file(json_path)?;
let sys_title = article.sys_title.clone();
// For simplicity, rebuild the entire store.
// A future optimization can do incremental append + index update.
self.rebuild()?;
Ok(sys_title)
}
/// Get the current version number for a sys_title.
pub fn article_version(&self, sys_title: &str) -> Option<u64> {
self.versions.get(sys_title).copied()
}
/// Get a reference to the content directory path.
pub fn content_dir(&self) -> &Path {
&self.content_dir
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
fn create_test_article(base: &Path, sys_title: &str, published: bool, tags: &[&str], ts: i64) {
let dir = base.join(sys_title);
fs::create_dir_all(&dir).unwrap();
let tags_json: Vec<String> = tags.iter().map(|t| format!("\"{}\"", t)).collect();
let json = format!(
r#"{{
"title": "{}",
"sys_title": "{}",
"published": {},
"content": {{
"author": "Author",
"content": {{
"sections": [{{ "heading": "Intro", "paragraphs": ["Hello."] }}],
"codes": [],
"images": []
}},
"tags": [{}],
"publishedOn": {}
}},
"published_on": {}
}}"#,
sys_title,
sys_title,
published,
tags_json.join(", "),
ts,
ts
);
fs::write(dir.join(format!("{}.json", sys_title)), json).unwrap();
}
#[test]
fn cold_start_and_queries() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let data = tmp.path().join("data");
create_test_article(&content, "art-one", true, &["rust"], 1000);
create_test_article(&content, "art-two", true, &["rust", "linux"], 2000);
create_test_article(&content, "art-three", false, &["draft"], 3000);
create_test_article(&content, "art-four", true, &["linux"], 4000);
let store = Store::open(&content, &data).unwrap();
// Title lookup.
let art = store.get_by_title("art-one").unwrap().unwrap();
assert_eq!(art.sys_title, "art-one");
assert!(store.get_by_title("nonexistent").unwrap().is_none());
// Tag query.
let rust_articles = store.list_by_tag("rust").unwrap();
assert_eq!(rust_articles.len(), 2);
let linux_articles = store.list_by_tag("linux").unwrap();
assert_eq!(linux_articles.len(), 2);
// Date range.
let range = store.list_by_date_range(1500, 3500).unwrap();
assert_eq!(range.len(), 2); // art-two(2000) and art-three(3000)
// Published count.
let count = store.count_published().unwrap();
assert_eq!(count, 3);
// Version tracking.
assert!(store.article_version("art-one").is_some());
assert!(store.article_version("nonexistent").is_none());
}
#[test]
fn rebuild_after_delete() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let data = tmp.path().join("data");
create_test_article(&content, "art-one", true, &["test"], 1000);
// First open: cold start.
let store = Store::open(&content, &data).unwrap();
assert_eq!(store.count_published().unwrap(), 1);
drop(store);
// Delete data/ and reopen: should rebuild.
fs::remove_dir_all(&data).unwrap();
let store = Store::open(&content, &data).unwrap();
assert_eq!(store.count_published().unwrap(), 1);
}
#[test]
fn reopen_existing() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let data = tmp.path().join("data");
create_test_article(&content, "art-one", true, &["test"], 1000);
// First open: cold start builds .seg + indexes.
let store = Store::open(&content, &data).unwrap();
assert_eq!(store.count_published().unwrap(), 1);
drop(store);
// Second open: loads existing files, no rebuild.
let store = Store::open(&content, &data).unwrap();
assert_eq!(store.count_published().unwrap(), 1);
let art = store.get_by_title("art-one").unwrap().unwrap();
assert_eq!(art.sys_title, "art-one");
}
}

16
reference/crates/wo-sub/Cargo.toml
View file

@ -1,16 +0,0 @@
[package]
name = "wo-sub"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-model = { path = "../wo-model" }
wo-store = { path = "../wo-store" }
wo-event = { path = "../wo-event" }
libc = { workspace = true }
bincode = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
[dev-dependencies]
tempfile = "3"

470
reference/crates/wo-sub/src/lib.rs
View file

@ -1,470 +0,0 @@
use std::collections::HashMap;
use std::io;
use std::os::unix::io::RawFd;
use serde::{Deserialize, Serialize};
use wo_event::EventFd;
/// A subscription query pattern.
#[derive(Debug, Clone, PartialEq)]
pub enum Subscription {
/// Subscribe to changes for a single article by sys_title.
ByTitle(String),
/// Subscribe to changes for all articles with a given tag.
ByTag(String),
/// Subscribe to all content changes.
All,
}
/// The type of content change.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub enum ChangeType {
Created,
Updated,
Deleted,
}
/// Notification payload written to subscriber fds.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub struct Notification {
pub change_type: ChangeType,
pub sys_title: String,
pub version: u64,
}
/// Wire format for notification payloads.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum FdFormat {
/// Length-prefixed bincode (for internal subscribers).
Binary,
/// Length-prefixed JSON (for browser socket fds).
Json,
}
/// Manages subscriptions and delivers notifications to subscriber fds.
///
/// Subscribers register interest via `subscribe()` with a query pattern.
/// When content changes, `notify()` evaluates which subscriptions match
/// and writes a length-prefixed notification to each matching fd.
pub struct SubscriptionManager {
/// sys_title → list of subscriber fds.
by_title: HashMap<String, Vec<RawFd>>,
/// tag → list of subscriber fds.
by_tag: HashMap<String, Vec<RawFd>>,
/// Fds subscribed to all changes.
global: Vec<RawFd>,
/// fd → list of subscriptions (for cleanup on disconnect).
fd_registry: HashMap<RawFd, Vec<Subscription>>,
/// fd → wire format.
fd_format: HashMap<RawFd, FdFormat>,
/// EventFd for signaling the event loop that notifications are pending.
notify_efd: EventFd,
}
impl SubscriptionManager {
/// Create a new subscription manager.
pub fn new() -> io::Result<Self> {
Ok(Self {
by_title: HashMap::new(),
by_tag: HashMap::new(),
global: Vec::new(),
fd_registry: HashMap::new(),
fd_format: HashMap::new(),
notify_efd: EventFd::new()?,
})
}
/// The eventfd for signaling the event loop.
pub fn notify_fd(&self) -> RawFd {
self.notify_efd.fd()
}
/// Register a subscription for the given fd (binary format).
pub fn subscribe(&mut self, fd: RawFd, query: Subscription) {
self.subscribe_with_format(fd, query, FdFormat::Binary);
}
/// Register a subscription for the given fd (JSON format for browsers).
pub fn subscribe_json(&mut self, fd: RawFd, query: Subscription) {
self.subscribe_with_format(fd, query, FdFormat::Json);
}
fn subscribe_with_format(&mut self, fd: RawFd, query: Subscription, format: FdFormat) {
match &query {
Subscription::ByTitle(title) => {
self.by_title.entry(title.clone()).or_default().push(fd);
}
Subscription::ByTag(tag) => {
self.by_tag.entry(tag.clone()).or_default().push(fd);
}
Subscription::All => {
self.global.push(fd);
}
}
self.fd_registry.entry(fd).or_default().push(query);
self.fd_format.insert(fd, format);
}
/// Remove all subscriptions for a fd (e.g., on EPOLLHUP).
pub fn unsubscribe(&mut self, fd: RawFd) {
self.fd_format.remove(&fd);
if let Some(subs) = self.fd_registry.remove(&fd) {
for sub in subs {
match sub {
Subscription::ByTitle(title) => {
if let Some(fds) = self.by_title.get_mut(&title) {
fds.retain(|&f| f != fd);
}
}
Subscription::ByTag(tag) => {
if let Some(fds) = self.by_tag.get_mut(&tag) {
fds.retain(|&f| f != fd);
}
}
Subscription::All => {
self.global.retain(|&f| f != fd);
}
}
}
}
}
/// Notify subscribers about a content change.
///
/// Evaluates which subscriptions match the changed sys_title and tags,
/// then writes a length-prefixed notification to each matching fd.
pub fn notify(
&self,
sys_title: &str,
tags: &[String],
change_type: ChangeType,
version: u64,
) -> io::Result<usize> {
let notification = Notification {
change_type,
sys_title: sys_title.to_string(),
version,
};
// Pre-serialize both formats (only actually used if there are subscribers).
let bincode_payload = bincode::serialize(&notification).map_err(|e| {
io::Error::new(io::ErrorKind::InvalidData, format!("bincode: {}", e))
})?;
let json_payload = serde_json::to_vec(&notification).map_err(|e| {
io::Error::new(io::ErrorKind::InvalidData, format!("json: {}", e))
})?;
// Build wire formats: [u32 length][payload]
let bincode_wire = build_wire(&bincode_payload);
let json_wire = build_wire(&json_payload);
// Collect all fds that should receive this notification.
let mut target_fds: Vec<RawFd> = Vec::new();
// Title subscribers.
if let Some(fds) = self.by_title.get(sys_title) {
target_fds.extend(fds);
}
// Tag subscribers.
for tag in tags {
if let Some(fds) = self.by_tag.get(tag) {
target_fds.extend(fds);
}
}
// Global subscribers.
target_fds.extend(&self.global);
// Deduplicate.
target_fds.sort_unstable();
target_fds.dedup();
let mut sent = 0;
for &fd in &target_fds {
let format = self.fd_format.get(&fd).copied().unwrap_or(FdFormat::Binary);
let wire = match format {
FdFormat::Binary => &bincode_wire,
FdFormat::Json => &json_wire,
};
if write_all_fd(fd, wire).is_ok() {
sent += 1;
}
}
// Signal the event loop.
if sent > 0 {
let _ = self.notify_efd.write(1);
}
Ok(sent)
}
/// Number of active subscriber fds.
pub fn subscriber_count(&self) -> usize {
self.fd_registry.len()
}
}
fn build_wire(payload: &[u8]) -> Vec<u8> {
let len = payload.len() as u32;
let mut wire = Vec::with_capacity(4 + payload.len());
wire.extend_from_slice(&len.to_le_bytes());
wire.extend_from_slice(payload);
wire
}
/// Write all bytes to a raw fd.
fn write_all_fd(fd: RawFd, buf: &[u8]) -> io::Result<()> {
let mut written = 0;
while written < buf.len() {
let ret = unsafe {
libc::write(
fd,
buf[written..].as_ptr() as *const libc::c_void,
buf.len() - written,
)
};
if ret < 0 {
return Err(io::Error::last_os_error());
}
if ret == 0 {
return Err(io::Error::new(io::ErrorKind::WriteZero, "write returned 0"));
}
written += ret as usize;
}
Ok(())
}
/// Convenience macro for compile-time subscription wiring.
///
/// ```ignore
/// register!(sub_manager, fd, ByTitle("linux-misc"));
/// register!(sub_manager, fd, ByTag("rust"));
/// register!(sub_manager, fd, All);
/// ```
#[macro_export]
macro_rules! register {
($mgr:expr, $fd:expr, ByTitle($title:expr)) => {
$mgr.subscribe($fd, $crate::Subscription::ByTitle($title.into()))
};
($mgr:expr, $fd:expr, ByTag($tag:expr)) => {
$mgr.subscribe($fd, $crate::Subscription::ByTag($tag.into()))
};
($mgr:expr, $fd:expr, All) => {
$mgr.subscribe($fd, $crate::Subscription::All)
};
}
#[cfg(test)]
mod tests {
use super::*;
/// Create a pipe and return (read_fd, write_fd).
fn make_pipe() -> (RawFd, RawFd) {
let mut fds = [0i32; 2];
assert_eq!(unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_NONBLOCK) }, 0);
(fds[0], fds[1])
}
fn read_notification(fd: RawFd) -> Notification {
let mut len_buf = [0u8; 4];
let ret = unsafe { libc::read(fd, len_buf.as_mut_ptr() as *mut libc::c_void, 4) };
assert!(ret == 4, "failed to read length prefix");
let len = u32::from_le_bytes(len_buf) as usize;
let mut payload = vec![0u8; len];
let ret = unsafe { libc::read(fd, payload.as_mut_ptr() as *mut libc::c_void, len) };
assert_eq!(ret as usize, len);
bincode::deserialize(&payload).unwrap()
}
fn close_fd(fd: RawFd) {
unsafe { libc::close(fd) };
}
#[test]
fn title_subscription() {
let mut mgr = SubscriptionManager::new().unwrap();
let (r1, w1) = make_pipe();
let (r2, w2) = make_pipe();
mgr.subscribe(w1, Subscription::ByTitle("art-one".into()));
mgr.subscribe(w2, Subscription::ByTitle("art-one".into()));
let sent = mgr
.notify("art-one", &[], ChangeType::Updated, 1)
.unwrap();
assert_eq!(sent, 2);
let n1 = read_notification(r1);
assert_eq!(n1.sys_title, "art-one");
assert_eq!(n1.change_type, ChangeType::Updated);
assert_eq!(n1.version, 1);
let n2 = read_notification(r2);
assert_eq!(n2.sys_title, "art-one");
// Unrelated title: no notifications.
let sent = mgr
.notify("art-two", &[], ChangeType::Created, 2)
.unwrap();
assert_eq!(sent, 0);
close_fd(r1); close_fd(w1);
close_fd(r2); close_fd(w2);
}
#[test]
fn tag_subscription() {
let mut mgr = SubscriptionManager::new().unwrap();
let (r, w) = make_pipe();
mgr.subscribe(w, Subscription::ByTag("rust".into()));
let sent = mgr
.notify("art-one", &["rust".into(), "linux".into()], ChangeType::Updated, 1)
.unwrap();
assert_eq!(sent, 1);
let n = read_notification(r);
assert_eq!(n.sys_title, "art-one");
// No "rust" tag: no notification.
let sent = mgr
.notify("art-two", &["python".into()], ChangeType::Created, 2)
.unwrap();
assert_eq!(sent, 0);
close_fd(r); close_fd(w);
}
#[test]
fn global_subscription() {
let mut mgr = SubscriptionManager::new().unwrap();
let (r, w) = make_pipe();
mgr.subscribe(w, Subscription::All);
let sent = mgr.notify("anything", &[], ChangeType::Deleted, 5).unwrap();
assert_eq!(sent, 1);
let n = read_notification(r);
assert_eq!(n.change_type, ChangeType::Deleted);
close_fd(r); close_fd(w);
}
#[test]
fn unsubscribe_removes_fd() {
let mut mgr = SubscriptionManager::new().unwrap();
let (r, w) = make_pipe();
mgr.subscribe(w, Subscription::ByTitle("art-one".into()));
mgr.subscribe(w, Subscription::All);
assert_eq!(mgr.subscriber_count(), 1);
mgr.unsubscribe(w);
assert_eq!(mgr.subscriber_count(), 0);
let sent = mgr.notify("art-one", &[], ChangeType::Updated, 1).unwrap();
assert_eq!(sent, 0);
close_fd(r); close_fd(w);
}
#[test]
fn dedup_notifications() {
let mut mgr = SubscriptionManager::new().unwrap();
let (r, w) = make_pipe();
// Same fd subscribed to both title AND tag — should only get one notification.
mgr.subscribe(w, Subscription::ByTitle("art-one".into()));
mgr.subscribe(w, Subscription::ByTag("rust".into()));
let sent = mgr
.notify("art-one", &["rust".into()], ChangeType::Updated, 1)
.unwrap();
assert_eq!(sent, 1); // deduped
let n = read_notification(r);
assert_eq!(n.sys_title, "art-one");
close_fd(r); close_fd(w);
}
#[test]
fn json_format_subscription() {
let mut mgr = SubscriptionManager::new().unwrap();
let (r, w) = make_pipe();
mgr.subscribe_json(w, Subscription::ByTitle("art-one".into()));
mgr.notify("art-one", &[], ChangeType::Updated, 1).unwrap();
// Read length-prefixed JSON.
let mut len_buf = [0u8; 4];
let ret = unsafe { libc::read(r, len_buf.as_mut_ptr() as *mut libc::c_void, 4) };
assert_eq!(ret, 4);
let len = u32::from_le_bytes(len_buf) as usize;
let mut payload = vec![0u8; len];
let ret = unsafe { libc::read(r, payload.as_mut_ptr() as *mut libc::c_void, len) };
assert_eq!(ret as usize, len);
// Should be valid JSON.
let json: serde_json::Value = serde_json::from_slice(&payload).unwrap();
assert_eq!(json["sys_title"], "art-one");
assert_eq!(json["change_type"], "Updated");
assert_eq!(json["version"], 1);
close_fd(r); close_fd(w);
}
#[test]
fn mixed_format_subscribers() {
let mut mgr = SubscriptionManager::new().unwrap();
let (r_bin, w_bin) = make_pipe();
let (r_json, w_json) = make_pipe();
mgr.subscribe(w_bin, Subscription::ByTitle("art-one".into()));
mgr.subscribe_json(w_json, Subscription::ByTitle("art-one".into()));
let sent = mgr.notify("art-one", &[], ChangeType::Created, 5).unwrap();
assert_eq!(sent, 2);
// Binary subscriber gets bincode.
let n = read_notification(r_bin);
assert_eq!(n.sys_title, "art-one");
assert_eq!(n.version, 5);
// JSON subscriber gets JSON.
let mut len_buf = [0u8; 4];
unsafe { libc::read(r_json, len_buf.as_mut_ptr() as *mut libc::c_void, 4) };
let len = u32::from_le_bytes(len_buf) as usize;
let mut payload = vec![0u8; len];
unsafe { libc::read(r_json, payload.as_mut_ptr() as *mut libc::c_void, len) };
let json: serde_json::Value = serde_json::from_slice(&payload).unwrap();
assert_eq!(json["sys_title"], "art-one");
close_fd(r_bin); close_fd(w_bin);
close_fd(r_json); close_fd(w_json);
}
#[test]
fn register_macro() {
let mut mgr = SubscriptionManager::new().unwrap();
let (r, w) = make_pipe();
register!(mgr, w, ByTitle("test-article"));
register!(mgr, w, ByTag("rust"));
register!(mgr, w, All);
assert_eq!(mgr.subscriber_count(), 1);
close_fd(r); close_fd(w);
}
}

11
reference/crates/wo-watch/Cargo.toml
View file

@ -1,11 +0,0 @@
[package]
name = "wo-watch"
version = "0.1.0"
edition = "2021"
[dependencies]
wo-store = { path = "../wo-store" }
libc = { workspace = true }
[dev-dependencies]
tempfile = "3"

280
reference/crates/wo-watch/src/lib.rs
View file

@ -1,280 +0,0 @@
use std::collections::HashMap;
use std::io;
use std::os::unix::io::RawFd;
use std::path::{Path, PathBuf};
/// A content change detected by the watcher.
#[derive(Debug, Clone, PartialEq)]
pub enum ContentChange {
Created(String), // sys_title
Modified(String), // sys_title
Deleted(String), // sys_title
}
/// Watches a content directory for article file changes using inotify.
///
/// Expects the directory structure:
/// ```text
/// content_dir/
/// {sys_title}/
/// {sys_title}.json
/// ```
pub struct ContentWatcher {
inotify_fd: RawFd,
/// Maps inotify watch descriptor → directory path.
wd_to_path: HashMap<i32, PathBuf>,
/// Maps directory path → watch descriptor.
path_to_wd: HashMap<PathBuf, i32>,
content_dir: PathBuf,
}
const EVENT_MASK: u32 =
(libc::IN_CREATE | libc::IN_MODIFY | libc::IN_DELETE | libc::IN_MOVED_TO | libc::IN_MOVED_FROM)
as u32;
impl ContentWatcher {
/// Create a new watcher on the given content directory.
///
/// Adds inotify watches on the content dir itself and all subdirectories.
pub fn new(content_dir: &Path) -> io::Result<Self> {
let fd = unsafe { libc::inotify_init1(libc::IN_NONBLOCK | libc::IN_CLOEXEC) };
if fd < 0 {
return Err(io::Error::last_os_error());
}
let mut watcher = Self {
inotify_fd: fd,
wd_to_path: HashMap::new(),
path_to_wd: HashMap::new(),
content_dir: content_dir.to_path_buf(),
};
// Watch the root content directory (for new article dirs).
watcher.add_watch(content_dir)?;
// Watch each existing subdirectory.
if content_dir.exists() {
for entry in std::fs::read_dir(content_dir)? {
let entry = entry?;
if entry.path().is_dir() {
watcher.add_watch(&entry.path())?;
}
}
}
Ok(watcher)
}
/// The inotify file descriptor, for registration on an event loop.
pub fn fd(&self) -> RawFd {
self.inotify_fd
}
/// Read and process pending inotify events.
///
/// Returns a list of content changes. Call this when the event loop
/// signals that the inotify fd is readable.
pub fn process_events(&mut self) -> io::Result<Vec<ContentChange>> {
let mut buf = [0u8; 4096];
let n = unsafe {
libc::read(
self.inotify_fd,
buf.as_mut_ptr() as *mut libc::c_void,
buf.len(),
)
};
if n < 0 {
let err = io::Error::last_os_error();
if err.raw_os_error() == Some(libc::EAGAIN) {
return Ok(vec![]);
}
return Err(err);
}
let mut changes = Vec::new();
let mut offset = 0usize;
while offset < n as usize {
let event = unsafe { &*(buf.as_ptr().add(offset) as *const libc::inotify_event) };
let name_len = event.len as usize;
let name = if name_len > 0 {
let name_ptr = unsafe { buf.as_ptr().add(offset + std::mem::size_of::<libc::inotify_event>()) };
let name_bytes = unsafe { std::slice::from_raw_parts(name_ptr, name_len) };
let end = name_bytes.iter().position(|&b| b == 0).unwrap_or(name_len);
Some(String::from_utf8_lossy(&name_bytes[..end]).to_string())
} else {
None
};
offset += std::mem::size_of::<libc::inotify_event>() + name_len;
let Some(name) = name else { continue };
let mask = event.mask;
// Event on the root content directory: subdirectory created/deleted.
if let Some(dir_path) = self.wd_to_path.get(&event.wd) {
let dir_path = dir_path.clone();
if dir_path == self.content_dir {
let sub_path = self.content_dir.join(&name);
if (mask & libc::IN_CREATE as u32) != 0 || (mask & libc::IN_MOVED_TO as u32) != 0 {
if sub_path.is_dir() {
let _ = self.add_watch(&sub_path);
changes.push(ContentChange::Created(name.clone()));
}
}
if (mask & libc::IN_DELETE as u32) != 0 || (mask & libc::IN_MOVED_FROM as u32) != 0 {
self.remove_watch(&sub_path);
changes.push(ContentChange::Deleted(name));
}
} else {
// Event inside an article subdirectory.
if name.ends_with(".json") || name.ends_with(".md") {
let sys_title = dir_path
.file_name()
.unwrap()
.to_string_lossy()
.to_string();
if (mask & libc::IN_MODIFY as u32) != 0 {
changes.push(ContentChange::Modified(sys_title));
} else if (mask & libc::IN_CREATE as u32) != 0
|| (mask & libc::IN_MOVED_TO as u32) != 0
{
changes.push(ContentChange::Created(sys_title));
} else if (mask & libc::IN_DELETE as u32) != 0
|| (mask & libc::IN_MOVED_FROM as u32) != 0
{
changes.push(ContentChange::Deleted(sys_title));
}
}
}
}
}
// Deduplicate: keep only the last change per sys_title.
let mut seen = HashMap::new();
for change in changes {
let key = match &change {
ContentChange::Created(t) | ContentChange::Modified(t) | ContentChange::Deleted(t) => {
t.clone()
}
};
seen.insert(key, change);
}
Ok(seen.into_values().collect())
}
fn add_watch(&mut self, path: &Path) -> io::Result<()> {
let c_path = std::ffi::CString::new(path.to_str().unwrap_or("")).map_err(|_| {
io::Error::new(io::ErrorKind::InvalidInput, "invalid path for inotify")
})?;
let wd = unsafe { libc::inotify_add_watch(self.inotify_fd, c_path.as_ptr(), EVENT_MASK) };
if wd < 0 {
return Err(io::Error::last_os_error());
}
self.wd_to_path.insert(wd, path.to_path_buf());
self.path_to_wd.insert(path.to_path_buf(), wd);
Ok(())
}
fn remove_watch(&mut self, path: &Path) {
if let Some(wd) = self.path_to_wd.remove(path) {
unsafe { libc::inotify_rm_watch(self.inotify_fd, wd) };
self.wd_to_path.remove(&wd);
}
}
}
impl Drop for ContentWatcher {
fn drop(&mut self) {
unsafe { libc::close(self.inotify_fd) };
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
#[test]
fn detect_json_modification() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let art_dir = content.join("my-article");
fs::create_dir_all(&art_dir).unwrap();
fs::write(art_dir.join("my-article.json"), r#"{"test": true}"#).unwrap();
let mut watcher = ContentWatcher::new(&content).unwrap();
// Modify the file.
fs::write(art_dir.join("my-article.json"), r#"{"test": false}"#).unwrap();
// Give inotify a moment.
std::thread::sleep(std::time::Duration::from_millis(50));
let changes = watcher.process_events().unwrap();
assert!(!changes.is_empty());
assert!(changes.iter().any(|c| matches!(c, ContentChange::Modified(t) if t == "my-article")));
}
#[test]
fn detect_new_directory() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
fs::create_dir_all(&content).unwrap();
let mut watcher = ContentWatcher::new(&content).unwrap();
// Create a new article directory.
let art_dir = content.join("new-article");
fs::create_dir_all(&art_dir).unwrap();
std::thread::sleep(std::time::Duration::from_millis(50));
let changes = watcher.process_events().unwrap();
assert!(changes.iter().any(|c| matches!(c, ContentChange::Created(t) if t == "new-article")));
}
#[test]
fn detect_json_delete() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let art_dir = content.join("delete-me");
fs::create_dir_all(&art_dir).unwrap();
fs::write(art_dir.join("delete-me.json"), "{}").unwrap();
let mut watcher = ContentWatcher::new(&content).unwrap();
// Delete the JSON file.
fs::remove_file(art_dir.join("delete-me.json")).unwrap();
std::thread::sleep(std::time::Duration::from_millis(50));
let changes = watcher.process_events().unwrap();
assert!(changes.iter().any(|c| matches!(c, ContentChange::Deleted(t) if t == "delete-me")));
}
#[test]
fn ignore_non_json_files() {
let tmp = tempfile::tempdir().unwrap();
let content = tmp.path().join("content");
let art_dir = content.join("my-article");
fs::create_dir_all(&art_dir).unwrap();
let mut watcher = ContentWatcher::new(&content).unwrap();
// Create a non-JSON file.
fs::write(art_dir.join("notes.txt"), "not json").unwrap();
std::thread::sleep(std::time::Duration::from_millis(50));
let changes = watcher.process_events().unwrap();
// Non-JSON files should not trigger Modified events.
assert!(changes.iter().all(|c| !matches!(c, ContentChange::Modified(_))));
}
}

92
reference/rest/README.md
View file

@ -1,92 +0,0 @@
# `reference/rest/` — HTTP test files for the `.wo` runtime
`.rest` (or `.http`) is the plain-text HTTP-request format supported by the two main editor HTTP clients:
- **VS Code** — install [REST Client](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) (`humao.rest-client`) and click "Send Request" above any block.
- **JetBrains IDEs** (IntelliJ, WebStorm, RustRover, Goland) — built-in HTTP Client recognises `.rest` and `.http` natively.
Both clients understand:
- `### ...` block separators
- `@var = value` document-level variables referenced as `{{var}}`
- `# @name foo` on a request, whose response fields are later reachable as `{{foo.response.body.id}}` — useful for threading auto-generated ids from `create` responses into later `get`/`patch`/`delete` calls
## Files
| File | Against | What it exercises |
| --- | --- | --- |
| [`blog.rest`](./blog.rest) | `docs/examples/blog/` | Full CRUD on Article/Comment, read-only on Author/Tag (matches the sample's `expose` lists). End-to-end flow: create article → list → get by id → PATCH title → PATCH embedded doc → DELETE draft → verify final state. |
| [`ecommerce.rest`](./ecommerce.rest) | `docs/examples/ecommerce/` | What Stage 2 currently serves for the ecommerce sample: read-only Product/Order/Customer lists, 405s for non-exposed create endpoints, 501s for Stage 3 stubs. Documents the shape of Stage 3/4 endpoints (`fn checkout`, LIVE subscribe, `/me`) even though they're not wired yet. |
## Running
```bash
# one terminal — start the runtime
cargo run --bin wo -- run docs/examples/blog
# [wo] listening on http://127.0.0.1:8080
# another terminal — or just open the .rest file in VS Code/JetBrains and click
```
Override the port via `WO_LISTEN`:
```bash
WO_LISTEN=127.0.0.1:9000 cargo run --bin wo -- run docs/examples/blog
```
…and update the `@host` line at the top of the `.rest` file to match.
## Without an editor (just `curl`)
Each `.rest` block maps directly to `curl`. Some examples:
```bash
# Runtime info
curl http://127.0.0.1:8080/
# List
curl http://127.0.0.1:8080/api/articles
# Create — server assigns `id` automatically
curl -X POST http://127.0.0.1:8080/api/articles \
-H "Content-Type: application/json" \
-d '{
"slug": "hello-writeonce",
"title": "Hello, writeonce",
"author": 1,
"published": true,
"meta": { "excerpt": "first post", "body_md": "# hi" }
}'
# Get by id
curl http://127.0.0.1:8080/api/articles/1
# Partial update
curl -X PATCH http://127.0.0.1:8080/api/articles/1 \
-H "Content-Type: application/json" \
-d '{ "title": "Updated" }'
# Delete
curl -X DELETE http://127.0.0.1:8080/api/articles/1 # 204 on success
# Stage 3 stub
curl -i http://127.0.0.1:8080/api/articles/live # 501 Not Implemented
```
For a scripted smoke run against the blog sample, the top-to-bottom `curl` sequence that exactly mirrors `blog.rest` is in [`docs/examples/blog/README.md`](../../docs/examples/blog/README.md).
## Expected-status cheat sheet
Every block in the `.rest` files ends its description with the expected HTTP status. Quick legend:
| Status | Meaning in this prototype |
| --- | --- |
| `200` | OK — list / get / update succeeded |
| `201` | Created — new row, `id` in the response body |
| `204` | No Content — delete succeeded |
| `400` | Bad JSON body |
| `404` | No such row, OR no method at all is attached to the path (e.g. `/api/customers` when the `service rest` block doesn't `expose` any collection-root op) |
| `405` | Method not allowed — the path is registered for a *different* method (e.g. POST against `/api/products` when only `list` is exposed, so GET is attached but POST isn't) |
| `501` | Not Implemented — Stage 3+ feature (LIVE subscriptions, `/me`, transactional fns) |
A `405` is a *feature* of the sample — it confirms the `expose` list in the `.wo` file is being honoured. A `501` is a Stage marker — the runtime acknowledges the shape but hasn't wired the handler yet.

181
reference/rest/blog.rest
View file

@ -1,181 +0,0 @@
###############################################################################
# blog.rest — exercise the `.wo` runtime against docs/examples/blog/
#
# Start the server first:
# cargo run --bin wo -- run docs/examples/blog
#
# Then in VS Code (REST Client extension) or JetBrains (HTTP Client):
# click "Send Request" on each block, top to bottom
#
# Response back-references (# @name foo → {{foo.response.body.id}}) are
# supported by both clients — later blocks pick up ids minted by earlier
# blocks automatically. For `curl` equivalents, see reference/rest/README.md.
###############################################################################
@host = http://127.0.0.1:8080
### Runtime info — expected 200
GET {{host}}/
### Liveness probe — expected 200 "ok"
GET {{host}}/healthz
###############################################################################
# Author — exposes: list, get, me, subscribe
###############################################################################
### List authors (empty on fresh boot) — expected 200 []
GET {{host}}/api/authors
### Author create is NOT exposed in the blog sample (expose list, get, me, subscribe)
# Expected 405 — method not allowed. The sample expects authors seeded by
# `on startup do: seed_admin()` in app.wo. Stage 2 doesn't run startup hooks
# yet, so the list above will be empty.
POST {{host}}/api/authors
Content-Type: application/json
{ "email": "alice@example.com", "handle": "alice", "display": "Alice" }
### /me — Stage 3 session layer; expected 501
GET {{host}}/api/authors/me
### LIVE subscribe — Stage 3; expected 501
GET {{host}}/api/authors/live
###############################################################################
# Article — exposes: list, get, create, update, delete, subscribe
###############################################################################
### Create an article — expected 201
# @name createArticle
POST {{host}}/api/articles
Content-Type: application/json
{
"slug": "hello-writeonce",
"title": "Hello, writeonce",
"author": 1,
"published": true,
"meta": {
"excerpt": "First post on the new runtime.",
"body_md": "# Hi\n\nHello from the `.wo` runtime. The server, the database, and this HTTP API are all one binary.\n"
}
}
### Create a second article (draft) — expected 201
# @name createDraft
POST {{host}}/api/articles
Content-Type: application/json
{
"slug": "second-draft",
"title": "Second Post (draft)",
"author": 1,
"published": false,
"meta": { "excerpt": "", "body_md": "WIP." }
}
### List articles — expected 200 with 2 rows
GET {{host}}/api/articles
### Get one article by id — expected 200
GET {{host}}/api/articles/{{createArticle.response.body.id}}
### Partial update (PATCH) — change just the title — expected 200
PATCH {{host}}/api/articles/{{createArticle.response.body.id}}
Content-Type: application/json
{ "title": "Hi, writeonce!" }
### Partial update of an embedded-document field — expected 200
# Note: Stage 2's PATCH does a shallow merge at the top level.
# To change `meta.excerpt` alone you re-send the whole `meta` object.
PATCH {{host}}/api/articles/{{createArticle.response.body.id}}
Content-Type: application/json
{
"meta": { "excerpt": "Updated excerpt.", "body_md": "# Hi\n\nUpdated body." }
}
### Publish the draft — expected 200 (`on update` trigger that sets
### `published_at` is Stage 3+; Stage 2 just records the bool change).
PATCH {{host}}/api/articles/{{createDraft.response.body.id}}
Content-Type: application/json
{ "published": true }
### Delete the draft — expected 204 No Content
DELETE {{host}}/api/articles/{{createDraft.response.body.id}}
### Get the deleted id — expected 404
GET {{host}}/api/articles/{{createDraft.response.body.id}}
### LIVE subscribe (Article) — Stage 3; expected 501
GET {{host}}/api/articles/live
###############################################################################
# Tag — exposes: list, get, subscribe
###############################################################################
### List tags — expected 200 [] (create is not exposed)
GET {{host}}/api/tags
### Tag create NOT exposed (expose list, get, subscribe) — expected 405
POST {{host}}/api/tags
Content-Type: application/json
{ "slug": "rust", "label": "Rust" }
### LIVE subscribe (Tag) — Stage 3; expected 501
GET {{host}}/api/tags/live
###############################################################################
# Comment — exposes: list, get, create, update, delete, subscribe
###############################################################################
### Create a comment — expected 201
# @name createComment
POST {{host}}/api/comments
Content-Type: application/json
{
"article": {{createArticle.response.body.id}},
"author": 1,
"body": "Nice post. Runs on one binary which is still weird to me."
}
### List comments — expected 200 with 1 row
GET {{host}}/api/comments
### Get one comment — expected 200
GET {{host}}/api/comments/{{createComment.response.body.id}}
### Update comment body — expected 200 (the `on update when old.body != new.body
### do set self.edited_at = now()` trigger is Stage 3+; edited_at stays unset)
PATCH {{host}}/api/comments/{{createComment.response.body.id}}
Content-Type: application/json
{ "body": "Edited: really, one binary? Neat." }
### Delete comment — expected 204
DELETE {{host}}/api/comments/{{createComment.response.body.id}}
### LIVE subscribe (Comment) — Stage 3; expected 501
GET {{host}}/api/comments/live
###############################################################################
# Final state — should show the updated article still there, no drafts,
# no comments (all deleted above).
###############################################################################
### Final article list — expected 200 with 1 row
GET {{host}}/api/articles
### Final comment list — expected 200 []
GET {{host}}/api/comments

142
reference/rest/ecommerce.rest
View file

@ -1,142 +0,0 @@
###############################################################################
# ecommerce.rest — exercise the `.wo` runtime against docs/examples/ecommerce/
#
# Start the server first:
# cargo run --bin wo -- run docs/examples/ecommerce
#
# The ecommerce sample is heavier on features that land in later stages:
# * orders are minted by `fn checkout(customer, product, qty) in txn snapshot`
# — transactional functions are a Stage 3/4 addition. Stage 2 does not
# register `/api/fn/checkout` yet; the block below documents that.
# * customer/product/order rows are seeded by `on startup do: seed()` in
# app.wo — startup hooks are also Stage 3+. Lists start empty.
# * Order-status lifecycle triggers (`on update when old.status != Paid ...`)
# are Stage 3+.
#
# This file therefore focuses on what Stage 2 *does* serve — route wiring,
# empty-list reads, method-not-allowed for non-exposed operations, and the
# Stage-3 stubs that respond 501. It doubles as a living spec for what the
# ecommerce sample should behave like once Stage 3+ lands.
###############################################################################
@host = http://127.0.0.1:8080
### Runtime info — expected 200
GET {{host}}/
### Liveness probe — expected 200 "ok"
GET {{host}}/healthz
###############################################################################
# Product — exposes: list, get, subscribe
# Stage 2 does NOT expose create — admin console is expected to seed inventory.
###############################################################################
### List products — expected 200 [] (no startup seed yet)
GET {{host}}/api/products
### Product create NOT exposed — expected 405
POST {{host}}/api/products
Content-Type: application/json
{
"sku": "SKU-WIDGET",
"name": "Widget",
"price": 1999,
"meta": { "description": "A widget.", "images": [], "attributes": { "colour": "blue" } },
"inventory": { "on_hand": 50, "reserved": 0, "reorder_at": 10 }
}
### Get by id — expected 404 (nothing exists)
GET {{host}}/api/products/1
### LIVE subscribe — Stage 3; expected 501
GET {{host}}/api/products/live
###############################################################################
# Customer — exposes: get, me, update, subscribe
# Customer create is gated by admin/self-signup flows outside this sample's
# scope. For now the list endpoint is not exposed either.
###############################################################################
### List NOT exposed — expected 404
# Customer's `expose get, me, update, subscribe` has nothing at the collection
# root, so no route is registered at `/api/customers` at all. The server
# returns 404 "no route" rather than 405 "method not allowed".
GET {{host}}/api/customers
### Customer create NOT exposed — expected 404
# Same reason: no method attached to /api/customers.
POST {{host}}/api/customers
Content-Type: application/json
{ "email": "carol@shop.test", "name": "Carol", "role": "Customer" }
### Get by id — expected 404 (nothing exists)
GET {{host}}/api/customers/1
### Update by id — would work if the customer existed; expected 404
PATCH {{host}}/api/customers/1
Content-Type: application/json
{ "name": "Carol Updated" }
### /me — Stage 3 session layer; expected 501
GET {{host}}/api/customers/me
### LIVE subscribe — Stage 3; expected 501
GET {{host}}/api/customers/live
###############################################################################
# Order — exposes: list, get, subscribe
# Orders are created by `fn checkout(...)` (see logic/checkout.wo), not via
# POST. Stage 2 does not register transactional-fn endpoints, so the list is
# empty until Stage 3/4 brings them online.
###############################################################################
### List orders — expected 200 []
GET {{host}}/api/orders
### Order create NOT exposed (use fn checkout) — expected 405
POST {{host}}/api/orders
Content-Type: application/json
{ "customer": 1, "status": "Pending", "line_items": [] }
### LIVE subscribe — same WebSocket the `##ui #admin-orders` board opens in
### Stage 6. For now the endpoint responds 501. This is the single most
### requested Stage 3 endpoint for this sample.
GET {{host}}/api/orders/live
###############################################################################
# fn checkout — deferred to Stage 3/4 (transactional functions)
#
# When transactional-fn endpoints land, this block becomes the canonical
# cross-paradigm ACID test: one call updates the product's inventory doc,
# inserts an Order row, creates a Purchase graph edge, and threads the new
# order id through all three stores inside one BEGIN ... COMMIT.
#
# See docs/examples/ecommerce/logic/checkout.wo and
# docs/runtime/database/05-go-sdk.md "Checkout from Go without codegen".
###############################################################################
### Stage 2 returns 404 — the route isn't registered.
POST {{host}}/api/fn/checkout
Content-Type: application/json
{ "customer": 1, "product": 1, "qty": 2 }
###############################################################################
# Purchase (link type) — no `service rest` block declared
# Purchase edges are created by fn checkout and read via
# `Customer.purchased` traversal (Stage 3+ query language).
###############################################################################
### Purchase list NOT exposed — expected 404 (no route registered)
GET {{host}}/api/purchases

Some files were not shown because too many files have changed in this diff Show more