diff --git a/CLAUDE.md b/CLAUDE.md index 5856ac4..3fbdcdc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,7 +26,10 @@ The repo holds three cuts of the same project plus one research reference: 1. **`crates/rt/`** — the **active Rust runtime** (Stage 2 shipped). Monolithic on purpose for now: lexer, parser, AST, in-memory engine, axum REST server all in one crate. The `wo` binary lives at `crates/rt/src/bin/wo.rs`. 2. **`crates/{ql,value,engine,txn,db,wal,sub,http,gen,policy,logic,service,ui,app}/`** — 14 **empty sibling crates** scaffolded to match the 7-phase design. Each has a `Cargo.toml` + `src/lib.rs` with just a doc comment pointing at its phase spec. Real code moves in from `rt` as each phase activates; do NOT refactor `rt` to use these today — it would break Stage 2. 3. **`reference/crates/`** — the **v1 writeonce blog** (13 crates: `wo-seg`, `wo-index`, `wo-store`, `wo-htmlx`, etc.). This is a **nested Cargo workspace**, deliberately excluded from the root workspace. The v1 crates keep their `wo-` prefix; the new runtime crates dropped theirs. `cd reference/crates && cargo build` builds v1 standalone. See `docs/runtime/database/07-wo-seg-migration.md` for the plan replacing v1 with the new runtime. -4. **`reference/linux/`** — **symlink to the Linux kernel source tree** (`/home/shoney/projects/linux`). Not committed (see `.gitignore`). Research resource for the kernel-primitives work: read `io_uring/`, `fs/notify/inotify/`, `kernel/eventfd.c`, `include/uapi/linux/*.h` when designing the runtime's kernel-facing modules. Each contributor sets their own target via `ln -s reference/linux`. +4. **`reference/linux/`** and **`reference/go/`** — **symlinks** to the Linux kernel source tree and the Go source tree respectively. Not committed (see `.gitignore`). Research resources: + - **`reference/linux/`** — grep `io_uring/`, `fs/notify/inotify/`, `kernel/eventfd.c`, `include/uapi/linux/*.h` when designing the kernel-primitive modules. Paired with per-primitive reference cards at [`docs/plan/linux/`](docs/plan/linux/). + - **`reference/go/`** — grep `src/runtime/netpoll_epoll.go`, `netpoll.go`, `os_linux*.go`, `asm_*.s`, `sys_linux_*.s` when designing the runtime layer. The `crates/rt/src/runtime/` module mirrors Go's `src/runtime/` file-per-flavour naming (`netpoll_epoll.rs` ↔ `netpoll_epoll.go`). The [`docs/plan/assembly/`](docs/plan/assembly/) docs cite this tree. + - Each contributor sets their own targets via `ln -s reference/{linux,go}`. There is also **`prototypes/wo-db/`** — a ~2k-line **C++ prototype** of the query-layer engine (SQL + Cypher + document paths, `RETURNING` aliases, `LIVE` stub). It keeps its `wo-db` directory name (C++ project, separate from the Rust crate `db`). It's the reference implementation the Rust port follows; `make test` still passes. @@ -134,5 +137,7 @@ Stage-3 stubs (501) and policy-shaped 405/404 responses are **intentional and do - `prototypes/wo-db/README.md` — the C++ prototype that shows the query layer - `reference/rest/README.md` — how to exercise the running prototype - `reference/README.md` — what's in the v1 archive and why it's preserved -- `reference/linux/` (symlink) — the Linux kernel source tree itself; grep `io_uring/`, `fs/notify/inotify/`, `include/uapi/linux/*.h` when designing kernel-facing modules +- `reference/linux/` (symlink) — the Linux kernel source tree; grep `io_uring/`, `fs/notify/inotify/`, `include/uapi/linux/*.h` when designing kernel-facing modules +- `reference/go/` (symlink) — the Go source tree; grep `src/runtime/netpoll_*.go`, `asm_*.s`, `sys_linux_*.s` when porting runtime-layer ideas (writeonce's `crates/rt/src/runtime/` mirrors this naming) +- `docs/plan/assembly/00-overview.md` — role of assembly in a runtime; writeonce policy is "no custom asm, use Rust stdlib" - `crates/README.md` — inventory of all 15 crates with phase assignments diff --git a/docs/plan/02-event-loop-epoll.md b/docs/plan/02-event-loop-epoll.md index 5b7d501..237e004 100644 --- a/docs/plan/02-event-loop-epoll.md +++ b/docs/plan/02-event-loop-epoll.md @@ -13,22 +13,26 @@ Nothing is removed in this phase. The module sits alongside the tokio-backed axu 1. **`epoll`, not `io_uring`, on day one.** `epoll` is ubiquitous (Linux 2.6+), well-understood, and every primitive we need (eventfd, timerfd, signalfd, inotify, accepted sockets) already integrates with it via `epoll_ctl`. `io_uring` is a natural follow-on phase once the event-loop abstraction exists — [`00-linux.md`](./linux/00-linux.md) calls it out for that role. 2. **Single-threaded, edge-triggered.** Matches [02-wo-language.md § Concurrency Model](../runtime/database/02-wo-language.md#concurrency-model). Every fd registered with `EPOLLET`; the loop reads until `EAGAIN`. No worker pool, no cross-thread state. 3. **`libc` is the only new dependency.** `libc = "0.2"` added to `crates/rt/Cargo.toml`. No `nix`, no `mio`. Direct `unsafe extern "C"` calls against the kernel surface. -4. **Module, not crate (yet).** Lives at `crates/rt/src/event/` so phase 03 can call into it cheaply. Extraction to the empty `crates/event/` sibling is deferred until a second caller appears outside `rt` — likely when [`sub`](../../crates/sub/) starts consuming the loop for subscription delivery. +4. **Module, not crate (yet).** Lives at `crates/rt/src/runtime/` so phase 03 can call into it cheaply. Extraction to the empty `crates/event/` sibling is deferred until a second caller appears outside `rt` — likely when [`sub`](../../crates/sub/) starts consuming the loop for subscription delivery. ## Scope -### New files inside `crates/rt/src/event/` +### New files inside `crates/rt/src/runtime/` | File | Responsibility | Port source | | --- | --- | --- | | `mod.rs` | Re-exports `EventLoop`, `Event`, `Interest`, `Token`, `EventFd`, `TimerFd`, `SignalFd` | [`reference/crates/wo-event/src/lib.rs`](../../reference/crates/wo-event/src/lib.rs) (9 LOC) | -| `epoll.rs` | `EventLoop { fd, events }` — `new()`, `register(raw_fd, interest, token)`, `wait_once(timeout) -> &[Event]`, `deregister(raw_fd)` | [`reference/crates/wo-event/src/epoll.rs`](../../reference/crates/wo-event/src/epoll.rs) (183 LOC) | +| `netpoll_epoll.rs` | `EventLoop { fd, events }` — `new()`, `register(raw_fd, interest, token)`, `wait_once(timeout) -> &[Event]`, `deregister(raw_fd)` | [`reference/crates/wo-event/src/epoll.rs`](../../reference/crates/wo-event/src/epoll.rs) (183 LOC); [`reference/go/src/runtime/netpoll_epoll.go`](../../reference/go/src/runtime/netpoll_epoll.go) for idiom | | `eventfd.rs` | `EventFd { fd }` — counter semaphore for cross-fd wake-up (subscription dispatch, shutdown signal) | [`reference/crates/wo-event/src/eventfd.rs`](../../reference/crates/wo-event/src/eventfd.rs) (66 LOC) | | `timerfd.rs` | `TimerFd { fd }` — oneshot + periodic timers as fds for the loop | [`reference/crates/wo-event/src/timerfd.rs`](../../reference/crates/wo-event/src/timerfd.rs) (91 LOC) | | `signalfd.rs` | `SignalFd { fd }` — SIGINT / SIGTERM / SIGHUP delivered as fd reads for graceful shutdown without a tokio signal handler | [`reference/crates/wo-event/src/signalfd.rs`](../../reference/crates/wo-event/src/signalfd.rs) (62 LOC) | Total: ~410 LOC lifted and adapted. The v1 code already compiles standalone in `reference/crates/wo-event/` and has unit tests; the port is near-verbatim plus namespace cleanups. +### Why `runtime/` not `event/` + +Go's equivalent code lives at [`reference/go/src/runtime/netpoll_epoll.go`](../../reference/go/src/runtime/netpoll_epoll.go) alongside siblings like `netpoll_kqueue.go` (macOS/BSD), `netpoll_io_uring.go` (if/when Go adds it), and the shared `netpoll.go` interface. The directory name "runtime" signals that this is the layer beneath user code — scheduler / netpoll / syscall shims — and the filename prefix `netpoll_` makes each implementation alternative visible at a glance. Adopting the same convention in writeonce makes porting ideas bidirectional: a reader who knows Go's layout can find the writeonce equivalent by trimming the `.go` extension and swapping it for `.rs`. When Phase 3's io_uring arrives it'll land as `netpoll_io_uring.rs` next to the epoll one; a cross-platform stub would be `netpoll.rs`. Module boundary and naming both match. See [`docs/plan/assembly/00-overview.md`](./assembly/00-overview.md) for why we stop short of mirroring Go's assembly conventions. + ### `Cargo.toml` change ```toml @@ -45,7 +49,7 @@ libc = "0.2" # NEW — see docs/plan/02-event-loop-epoll.md ## API shape (target — validate against v1 when porting) ```rust -use rt::event::{EventLoop, EventFd, Interest, Token}; +use rt::runtime::{EventLoop, EventFd, Interest, Token}; let mut loop_ = EventLoop::new()?; let ev = EventFd::new()?; @@ -62,7 +66,7 @@ for event in loop_.wait_once(Some(Duration::from_millis(100)))? { ## Exit criteria 1. `cargo build` at root compiles cleanly. -2. A new unit test in `crates/rt/src/event/epoll.rs`: +2. A new unit test in `crates/rt/src/runtime/netpoll_epoll.rs`: - create an `EventLoop`, - register an `EventFd`, - `write(1)` to the eventfd from the same thread, @@ -77,14 +81,14 @@ for event in loop_.wait_once(Some(Duration::from_millis(100)))? { - **No cutover.** The `wo` binary keeps calling `tokio::runtime::Builder::new_current_thread()`. That happens in phase 04. - **No HTTP.** Accepting connections is phase 03's problem. This phase is pure kernel-primitive plumbing. - **No subscription dispatch.** The `sub` crate doesn't exist yet as real code; phase 07 (inotify) is the first real loop consumer after phase 03. -- **No crate extraction.** Stays at `crates/rt/src/event/`. Pulling to `crates/event/` waits for a second consumer. +- **No crate extraction.** Stays at `crates/rt/src/runtime/`. Pulling to `crates/event/` waits for a second consumer. - **No `io_uring`.** Separate follow-on once the abstraction solidifies. ## Verification ```bash cargo build -cargo test --lib event # new tests in crates/rt/src/event/ +cargo test --lib runtime # new tests in crates/rt/src/runtime/ cargo test --lib # all 14 existing + new epoll/eventfd/timerfd tests green cargo run --bin wo -- run docs/examples/blog # axum path unchanged, still serves cd reference/crates && cargo build && cargo test # v1 untouched diff --git a/docs/plan/03-hand-rolled-http.md b/docs/plan/03-hand-rolled-http.md index 3cbe79c..6482378 100644 --- a/docs/plan/03-hand-rolled-http.md +++ b/docs/plan/03-hand-rolled-http.md @@ -12,7 +12,7 @@ A non-blocking HTTP/1.1 server module that accepts connections, parses requests, 2. **Per-connection state machine.** Each accepted socket fd is registered on the event loop with its own `Connection { state: Reading | Writing | Idle, parser, pending_response }`. Edge-triggered `EPOLLIN`/`EPOLLOUT` drive state transitions. Matches [v1 wo-http](../../reference/crates/wo-http/src/connection.rs)'s model verbatim. 3. **Router is pattern-matched at registration.** `Router::new().route("/api/articles/:id", Method::GET, handler)` resolves to a trie at boot. Per-request dispatch is a single trie walk — no axum-style type-erased layers. 4. **Handlers are `fn(&Request, &Engine) -> Response`.** Synchronous. The single-threaded event loop means a handler blocking is a bug; each handler must be a pure transformation over engine state. -5. **Module, not crate (yet).** Lives at `crates/rt/src/http/` with the same "extract when a second consumer shows up" rule as phase 02. The eventual home is the empty [`crates/http/`](../../crates/http/) sibling — but not in this phase. +5. **Module, not crate (yet).** Lives at `crates/rt/src/http/` with the same "extract when a second consumer shows up" rule as phase 02. The eventual home is the empty [`crates/http/`](../../crates/http/) sibling — but not in this phase. Paired with [phase 02's `crates/rt/src/runtime/`](./02-event-loop-epoll.md) (Go-style naming — `netpoll_epoll.rs`, `eventfd.rs`, …) which this module depends on for the `EventLoop` + raw syscall shims. Go's `src/net/http/` and `src/runtime/` split is the layout precedent; see [`reference/go/src/net/http/`](../../reference/go/src/net/http/). ## Scope diff --git a/docs/plan/04-cutover-remove-tokio-axum.md b/docs/plan/04-cutover-remove-tokio-axum.md index f1f0066..2c9faa5 100644 --- a/docs/plan/04-cutover-remove-tokio-axum.md +++ b/docs/plan/04-cutover-remove-tokio-axum.md @@ -43,7 +43,7 @@ Three deps gone. Four remaining: `anyhow`, `serde`, `serde_json`, `libc`. ### Files deleted -- None. The phase-02 `event/` module and phase-03 `http/` module stay in place and now become the primary code path. +- None. The phase-02 `runtime/` module and phase-03 `http/` module stay in place and now become the primary code path. ## Handler signature change @@ -91,7 +91,7 @@ Twelve handlers total — one pair per `{list, get, create, update, delete}` × - **No JSON replacement.** `serde` + `serde_json` are still imported and used. Phase 05 removes them. - **No inotify / sendfile.** Stage 3 capabilities. Phases 07 and 08. - **No `io_uring`.** The `EventLoop` keeps using `epoll` here; swapping is a later phase. -- **No crate extraction.** `event/` and `http/` stay inside `crates/rt/src/`. The empty `crates/event/` and `crates/http/` sibling crates wait for second consumers. +- **No crate extraction.** `runtime/` and `http/` stay inside `crates/rt/src/`. The empty `crates/http/` sibling crate waits for a second consumer; `runtime/` doesn't have a sibling slot (it's the binary's private kernel-primitive layer, analogous to Go's `src/runtime/` staying internal to the toolchain). ## Risk diff --git a/docs/plan/assembly/00-overview.md b/docs/plan/assembly/00-overview.md new file mode 100644 index 0000000..71235b6 --- /dev/null +++ b/docs/plan/assembly/00-overview.md @@ -0,0 +1,43 @@ +# 00 — The role of assembly in a runtime + +Why does a runtime ship hand-written assembly at all? Three reasons — each one a place where a higher-level language literally cannot express the operation it needs, so the compiler is bypassed and machine instructions are written directly. Go's [`src/runtime/`](../../../reference/go/src/runtime/) is the canonical example; this doc names the three reasons and points at the Go files that embody each. + +## 1 — Operations that violate the language's own calling convention + +The biggest category. The language's calling convention — how arguments are passed, who saves which registers, how the stack grows — is the contract every compiled function obeys. A few runtime operations *have* to break it because they ARE the mechanism by which control flow enters and exits that contract. + +**Goroutine stack switching.** When Go's scheduler switches from one goroutine to another, it's literally rewriting the stack pointer mid-function — jumping from one goroutine's stack to another's. The language compiler can't emit this safely because every function assumes its stack is the one it got called on. See [`reference/go/src/runtime/asm_amd64.s`](../../../reference/go/src/runtime/asm_amd64.s) for `TEXT runtime·gogo(SB)`, `TEXT runtime·mcall(SB)`, `TEXT runtime·systemstack(SB)` — all unavoidable. + +**Signal-handler entry.** When a signal arrives, the kernel drops the process onto an alternate stack with preserved registers. Returning to normal code means restoring everything the handler touched plus switching stacks back. Go's `runtime·sigtramp` in [`reference/go/src/runtime/sys_linux_amd64.s`](../../../reference/go/src/runtime/sys_linux_amd64.s) handles this. + +**Cgo boundary crossing.** Calling C from Go means switching to the OS thread's "real" stack (C expects contiguous stacks; Go uses segmented). Going back means the inverse. Entirely asm-driven. + +## 2 — Architecture-specific instructions the compiler doesn't emit + +Atomics, memory barriers, and some hardware-accelerated primitives need specific instruction sequences. A compiler that sees `a = *b` can't know whether you wanted a relaxed load or an acquire fence without annotation — and the *right* instruction on x86 vs ARM vs RISC-V is different. + +**Atomic CAS / load-acquire / store-release.** On x86 it's `LOCK CMPXCHG`; on ARM it's `LDXR` / `STXR` with a retry loop; on RISC-V it's `LR.W.AQ` / `SC.W.RL`. Go emits these from [`reference/go/src/runtime/atomic_amd64.s`](../../../reference/go/src/runtime/atomic_amd64.s) (and its per-arch siblings) because a portable compiler can't. + +**Memory barriers.** `MFENCE`, `LFENCE`, `SFENCE` on x86; `DMB` / `DSB` / `ISB` on ARM. Used by Go's `publicationBarrier`, `procyield`, and friends. Per-arch asm files carry them. + +**Optimised `memmove` / `memequal` / `memclr`.** The compiler knows how to emit `rep movsb`, but a runtime sometimes ships a *better* version than the compiler's — wider vector loads, prefetch hints, alignment-aware loops. Go ships its own in [`asm_amd64.s`](../../../reference/go/src/runtime/asm_amd64.s) using AVX/SSE paths. + +## 3 — Syscall trampolines + +Every raw syscall to the kernel is an asm stub. The kernel expects arguments in specific registers (on x86_64: `rdi`, `rsi`, `rdx`, `r10`, `r8`, `r9`, with the syscall number in `rax`), a `syscall` instruction, and return-value unpacking from `rax` (including `-errno` convention). A high-level language's calling convention doesn't match that layout — you need a thin asm wrapper per syscall. + +See [`reference/go/src/runtime/sys_linux_amd64.s`](../../../reference/go/src/runtime/sys_linux_amd64.s) — 43 `TEXT` functions, one per syscall family: `runtime·write`, `runtime·read`, `runtime·futex`, `runtime·clone`, `runtime·rt_sigaction`, `runtime·rt_sigprocmask`, `runtime·rt_sigreturn`, `runtime·sched_yield`, `runtime·mmap`, `runtime·munmap`, `runtime·madvise`, `runtime·epollcreate1`, `runtime·epollctl`, `runtime·epollwait`, etc. + +Go does these in asm because it cannot rely on libc — Go's scheduler needs to enter/exit syscalls at exactly controlled points (`runtime·entersyscall`, `runtime·exitsyscall`) so the M (OS thread) can be parked or reused without losing the goroutine. Going through `libc::write` would sidestep the scheduler's accounting. + +## How writeonce differs + +**Rust + libc covers all three categories** for the specific workload the `rt` crate serves. No custom scheduler means no stack switching. `std::sync::atomic::*` emits the right arch-specific instructions per target. `libc::syscall(SYS_*, ...)` hits the kernel through glibc's own trampolines — we don't need our own because we don't need fine-grained control over scheduler park/unpark (there's no scheduler to park). `signalfd` (see [`../linux/04-signalfd.md`](../linux/04-signalfd.md)) makes signal-handler asm unnecessary. + +The next document, [`01-go-runtime-asm.md`](./01-go-runtime-asm.md), catalogues Go's asm in concrete detail. The one after that, [`02-writeonce-stance.md`](./02-writeonce-stance.md), spells out the policy: **no custom assembly in `crates/rt`** — and lists the three edge cases where a future profiling run might force the decision. + +## Reading order + +1. **This doc** — the abstract "why asm exists in runtimes." +2. [`01-go-runtime-asm.md`](./01-go-runtime-asm.md) — concrete Go inventory with reference paths. +3. [`02-writeonce-stance.md`](./02-writeonce-stance.md) — the writeonce policy + escape hatches. diff --git a/docs/plan/assembly/01-go-runtime-asm.md b/docs/plan/assembly/01-go-runtime-asm.md new file mode 100644 index 0000000..e6b3a9f --- /dev/null +++ b/docs/plan/assembly/01-go-runtime-asm.md @@ -0,0 +1,91 @@ +# 01 — Go's runtime assembly, catalogued + +The Go runtime ships ~72 `TEXT` functions in `asm_amd64.s` alone, ~43 in `sys_linux_amd64.s`, and per-architecture variants of both for `386`, `arm`, `arm64`, `loong64`, `mips(64)x`, `ppc64x`, `riscv64`, `s390x`, `wasm`. This doc inventories them by purpose so a reader can map each Go asm concern to the writeonce equivalent (spoiler: usually "Rust stdlib does it"). Follow-on reading: [`02-writeonce-stance.md`](./02-writeonce-stance.md). + +All paths are inside [`reference/go/src/runtime/`](../../../reference/go/src/runtime/). + +## Scheduler & stack switching — `asm_.s` + +One file per arch, everything that has to break Go's calling convention. The x86_64 version lives at [`asm_amd64.s`](../../../reference/go/src/runtime/asm_amd64.s). + +| Go symbol | What | +| --- | --- | +| `runtime·gogo(SB)` | Jump to a goroutine's saved program counter on its stack. The machine-level act of resuming a suspended goroutine. | +| `runtime·mcall(SB)` | Call a function on the `g0` stack (the OS-thread's real stack). Used by anything that may block indefinitely — the calling goroutine gets parked. | +| `runtime·systemstack(SB)` | Transient jump to `g0` for a single function (GC operations, scheduler code) and back. | +| `runtime·morestack(SB)` | Stack-growth trampoline. Preamble-injected by the compiler when a function's stack usage exceeds the current segment; calls back into Go to allocate more. | +| `runtime·asmcgocall(SB)` | Enter C code. Switches from Go stack to the OS thread's C stack, marshals arguments, handles re-entry if C calls back into Go. | +| `runtime·cgocallback(SB)` | Re-enter Go from C. The inverse of the above. | +| `runtime·memmove(SB)` | Optimised memcpy; falls into SSE/AVX paths for large regions. Competes with (and often beats) the C library's version on specific inputs. | +| `runtime·memequal(SB)` | Fixed-length memory compare; used by map key comparisons and interface equality. | +| `runtime·memclrNoHeapPointers(SB)` | Fast zero-fill for regions the GC doesn't need to scan. Autovectorised; avoids Go-level write-barrier overhead. | +| `runtime·jmpdefer(SB)` | `defer` unwinding. Manually adjusts the frame pointer to jump into a deferred function as if it had been called at a different site. | +| `runtime·asyncPreempt(SB)` | Preemption entry point (Go 1.14+). A signal handler rewrites the target goroutine's PC to point here; on resume it parks and runs the scheduler. | + +**Why it's asm:** every function above manipulates state (stack pointer, program counter, register contents) that Go's language semantics don't let you express. + +## Atomics & barriers — `internal/runtime/atomic/atomic_.s` + +Lives at [`internal/runtime/atomic/atomic_amd64.s`](../../../reference/go/src/internal/runtime/atomic/atomic_amd64.s) (and arch variants). Wrappers around arch-specific instructions: + +| Go symbol | x86 instruction | Purpose | +| --- | --- | --- | +| `·Load` / `·Loadp` / `·Load64` | `MOV` | Acquire load. On x86, plain `MOV` is already acquire-ordered; on ARM the same operation needs `LDR` + `DMB ISH`. | +| `·Store` / `·Store64` / `·StoreRel` | `XCHG` or `MOV` + `MFENCE` | Release store. | +| `·Xchg` / `·Xchg64` | `XCHG` | Atomic swap. | +| `·Xadd` / `·Xadd64` | `LOCK XADD` | Atomic fetch-and-add. | +| `·Cas` / `·Cas64` / `·Casp` | `LOCK CMPXCHG` | Compare-and-swap. | +| `·And` / `·Or` / `·And8` / `·Or8` | `LOCK AND` / `LOCK OR` | Atomic bit manipulation. | + +**Why it's asm:** the *instruction* per operation differs per architecture. Go abstracts them behind a consistent `sync/atomic` surface. + +## Syscall trampolines — `sys__.s` + +On Linux-x86_64 that's [`sys_linux_amd64.s`](../../../reference/go/src/runtime/sys_linux_amd64.s) — 43 `TEXT` functions. Each is a short wrapper: move args into the kernel's register layout, execute `SYSCALL`, convert `rax` into a Go return value + error. + +| Go symbol | Linux syscall | +| --- | --- | +| `runtime·write` / `runtime·write1` | `write(2)` | +| `runtime·read` / `runtime·pread` | `read(2)` / `pread64(2)` | +| `runtime·closefd` | `close(2)` | +| `runtime·open` | `openat(2)` | +| `runtime·futex` | `futex(2)` | +| `runtime·clone` | `clone(2)` — M (OS thread) creation | +| `runtime·rt_sigaction` / `runtime·rt_sigprocmask` | Signal plumbing | +| `runtime·sigreturn` | Return from signal handler | +| `runtime·sched_yield` | Voluntary preemption | +| `runtime·mmap` / `runtime·munmap` / `runtime·madvise` | Memory operations | +| `runtime·epollcreate1` / `runtime·epollctl` / `runtime·epollwait` | The epoll trio | +| `runtime·exit` / `runtime·exitThread` | Process / thread termination | + +**Why it's asm (in Go):** libc's syscall wrappers check for cancellation, TLS, errno state — overhead Go's scheduler can't afford between `entersyscall` and `exitsyscall`. Go needs precise control over when the goroutine is "in a syscall" so the M can be detached from the P. + +**Note:** writeonce **does** use libc's syscall wrappers — we don't have a scheduler to detach, so the overhead is irrelevant. See the stance doc for the full rationale. + +## Signal handling — `sigtramp` in `sys__.s` + +`runtime·sigtramp` (same file as the syscalls) is the entry point the kernel jumps to when a signal fires. It saves the pre-signal register state, switches to the `g0` stack, calls into `runtime.sigtrampgo`, then restores on return. Needs asm because the ABI between kernel and user signal handler is rigid and arch-specific. + +**Writeonce avoids this entirely** by using `signalfd` — signals become fd reads on the epoll loop, no handler ever runs. See [`../linux/04-signalfd.md`](../linux/04-signalfd.md). + +## Cgo bridge — `cgo__.s` + +Files like [`cgo/asm_amd64.s`](../../../reference/go/src/runtime/cgo/asm_amd64.s). Machine-code marshalling between Go's register convention and C's SysV AMD64 ABI. Needed because Go's calling convention uses stack slots differently from C's register passing. + +**Writeonce doesn't cross language boundaries** — Rust is the only language in the binary; `libc` is already in Rust's register convention via `extern "C"`. No cgo bridge needed. + +## Timers & monotime — `time__.s` + +Small files providing the absolute-minimum latency paths for `nanotime()` and friends. On modern Linux they use `vDSO` entries (`__vdso_clock_gettime`) that bypass the syscall boundary. + +**Writeonce uses `std::time::Instant::now()`** which internally invokes the same vDSO through glibc — already optimised, no added value in rolling our own. + +## ASAN / MSAN / race detector — `asan_.s`, `race_.s` + +Hookable entry points for sanitisers. Not relevant to writeonce. + +## What's NOT in asm + +Everything else in Go's runtime is plain Go: the scheduler's policy (`proc.go`), the garbage collector (`mgc.go` etc.), `netpoll` dispatch (`netpoll.go` — *Go code*; the platform-specific backends like `netpoll_epoll.go` are also pure Go that call into the asm `epollwait` trampoline). The asm is strictly the three categories in [`00-overview.md`](./00-overview.md): calling-convention-breaking operations, arch-specific instructions, and syscall trampolines. + +The three categories writeonce **also** needs a solution for — but writeonce gets all three from Rust stdlib + libc. The next doc catalogues those mappings. diff --git a/docs/plan/assembly/02-writeonce-stance.md b/docs/plan/assembly/02-writeonce-stance.md new file mode 100644 index 0000000..86b9383 --- /dev/null +++ b/docs/plan/assembly/02-writeonce-stance.md @@ -0,0 +1,74 @@ +# 02 — The writeonce stance on assembly + +**Policy: no custom assembly in `crates/rt`.** Use Rust stdlib + `libc` for every category Go's runtime solves with `.s` files. If a future profiling pass proves an asm optimisation is necessary, confine it to `crates/rt/src/runtime/asm/` with one file per use case and an `#[cfg(target_arch = "...")]` cover per file (the Go `asm_amd64.s` pattern, one-per-architecture). + +## Why the policy works + +Each Go asm category from [`01-go-runtime-asm.md`](./01-go-runtime-asm.md) maps to a Rust-stdlib equivalent that is already correct on every supported architecture: + +| Go asm need | What writeonce uses | Why it covers the gap | +| --- | --- | --- | +| Scheduler stack switching (`gogo`, `mcall`, `systemstack`) | — nothing — | Single-threaded event loop (see Phase 2 [Concurrency Model](../runtime/database/02-wo-language.md#concurrency-model)). No goroutines, no stack switching, no `g0`. | +| Preemption (`asyncPreempt`) | — nothing — | No preemption. Handlers run to completion on the single thread. | +| Atomic operations (`Load`, `Store`, `Cas`, `Xadd`, ...) | [`std::sync::atomic`](https://doc.rust-lang.org/std/sync/atomic/) | The compiler emits the right instruction per target — `LOCK CMPXCHG` on x86, `LDXR/STXR` on ARM, `LR.W/SC.W` on RISC-V. Ordering is in the type signature (`Ordering::Acquire`, `Release`, `SeqCst`). | +| Memory barriers (`MFENCE` etc.) | [`std::sync::atomic::fence(Ordering)`](https://doc.rust-lang.org/std/sync/atomic/fn.fence.html) | One call, one fence, arch-neutral. | +| `memmove` / `memequal` / `memclr` | [`core::ptr::copy`](https://doc.rust-lang.org/core/ptr/fn.copy.html), `<[T]>::copy_from_slice`, `==`, `[T]::fill(0)` | LLVM emits the same vectorised code Go's asm does, often better because it knows alignment statically. | +| Syscall trampolines | `libc::write(...)`, `libc::read(...)`, `libc::syscall(libc::SYS_io_uring_enter, ...)` | No scheduler means no "park the M across this syscall" concern. Plain libc is the right abstraction. | +| Signal handler entry (`sigtramp`) | `signalfd` (see [`../linux/04-signalfd.md`](../linux/04-signalfd.md)) | Signals become fd reads on the epoll loop. No user-side handler runs; no trampoline needed. | +| Cgo boundary | — nothing — | Rust is the only language. `extern "C"` handles libc in the compiler's own ABI pass; no asm shim. | +| `nanotime` (vDSO) | [`std::time::Instant::now()`](https://doc.rust-lang.org/std/time/struct.Instant.html) | Already goes through the vDSO via glibc. No added value in rolling our own. | + +Net result: the assembly layer that accounts for Go's runtime complexity collapses into Rust's standard library and a handful of `libc::*` calls. Zero hand-written `.s` files in the target architecture. + +## Edge cases where asm might come up + +Three situations where a future developer might reach for `std::arch::asm!` — and the first-line answer for each, which is always "try this Rust-stdlib path first." + +### 1. `io_uring` ring atomics + +The SQ and CQ rings need specific memory ordering around the `head`/`tail` indices: producer writes release-ordered, consumer reads acquire-ordered. The kernel and userland form a lockless SPSC queue across the mmap'd region. + +- **Rust answer:** `AtomicU32` at the ring-index offsets with explicit `Ordering::Acquire` / `Ordering::Release`. `fence(Ordering::SeqCst)` where the kernel ABI demands a full barrier. This is how `io-uring` (the community crate) implements it — in safe Rust. +- **When to escalate:** never, realistically. Kernel + Rust stdlib agree on x86-TSO / ARM-AcRel semantics. An asm deviation would be a bug. + +### 2. SIMD in the phase-05 JSON parser + +`simd-json` and `serde-json-core` both show that JSON parsing benefits significantly from SIMD byte scans (whitespace skip, quote/escape detection). + +- **Rust answer:** `core::arch::x86_64::*` intrinsics (`_mm256_cmpeq_epi8`, `_mm_movemask_epi8`) inside `#[cfg(target_feature = "avx2")]` blocks, with a scalar fallback. All safe-enough Rust — no `asm!` block needed. `std::simd` (nightly) is even cleaner when it stabilises. +- **When to escalate:** if a specific instruction sequence outperforms the compiler's codegen by a meaningful margin *under measurement*, wrap it in `asm!` inside a single `#[cfg(target_arch = "x86_64")]` helper. Today the compiler is almost always as good as handwritten for well-understood patterns. + +### 3. WAL group-commit sub-µs timing + +Phase 3's WAL loop batches commits and fsyncs them in one `io_uring` submission. If we ever need to shave nanoseconds off the per-commit path — spin-waiting on the `io_uring` CQE ring, say — the tightest loop might want specific instructions (`PAUSE` on x86, `WFE` on ARM). + +- **Rust answer:** [`std::hint::spin_loop()`](https://doc.rust-lang.org/std/hint/fn.spin_loop.html). The compiler emits `PAUSE` / `WFE` per target. Use `core::hint::black_box` to prevent compiler over-optimisation of measurement. +- **When to escalate:** when benchmarks show a specific hotspot the compiler is provably wrong about. Not before. + +## The escape hatch + +If all three defences above are exhausted — benchmarked, documented, reviewed — assembly goes in: + +``` +crates/rt/src/runtime/asm/ +├── mod.rs # re-exports; arch gating +├── memcpy_avx2_x86_64.rs # #[cfg(all(target_arch = "x86_64", target_feature = "avx2"))] +└── spin_pause_x86_64.rs # #[cfg(target_arch = "x86_64")] +``` + +Rules: + +1. **One concern per file.** No `asm.rs` catch-all. +2. **Arch-gated at the file level.** `#[cfg(target_arch = "...")]` at the top of the file covering the whole module. A non-x86_64 build sees an empty module, not missing symbols. +3. **Scalar fallback in `mod.rs`.** Every asm function has a pure-Rust sibling gated with the negation. Hosts that fail the `target_feature` check get the fallback transparently. +4. **Comment every instruction with the intent.** `lock xadd` is opaque without "// atomic RMW for SQ head advance." +5. **Benchmark in the commit message.** Numbers before and after, the compiler version that produced the delta, and the specific CPU model. + +No asm has been written under this policy yet. The expectation is it stays that way for the foreseeable future — the [`docs/plan/02-08`](../) sequence reaches feature parity with Go's runtime using exclusively the Rust-stdlib + libc path. + +## Cross-references + +- [`00-overview.md`](./00-overview.md) — why runtimes ever need asm at all (three categories). +- [`01-go-runtime-asm.md`](./01-go-runtime-asm.md) — Go's asm inventory, by file. +- [`../linux/04-signalfd.md`](../linux/04-signalfd.md) — the specific primitive that obviates Go's `sigtramp` asm. +- [`../02-event-loop-epoll.md`](../02-event-loop-epoll.md) — phase 02, where the `runtime/` module actually lands. diff --git a/docs/plan/linux/00-linux.md b/docs/plan/linux/00-linux.md index bd42b10..549ec25 100644 --- a/docs/plan/linux/00-linux.md +++ b/docs/plan/linux/00-linux.md @@ -140,3 +140,6 @@ register! { Each pattern resolves to a set of inotify watch descriptors. When the watched set changes (new article added that matches the filter), the subscription table updates automatically during re-indexing. +## Related: the assembly policy + +Every primitive above is reached via `libc::` or `libc::syscall(SYS_*, ...)` — no custom assembly. The reasoning lives in [`../assembly/`](../assembly/) — three files covering why runtimes use asm at all ([`00-overview.md`](../assembly/00-overview.md)), what Go's [`reference/go/src/runtime/*.s`](../../../reference/go/src/runtime/) actually contains ([`01-go-runtime-asm.md`](../assembly/01-go-runtime-asm.md)), and the writeonce policy that all of it is replaced by Rust stdlib + libc ([`02-writeonce-stance.md`](../assembly/02-writeonce-stance.md)). diff --git a/reference/README.md b/reference/README.md index ed94fb8..6afd3fa 100644 --- a/reference/README.md +++ b/reference/README.md @@ -33,3 +33,15 @@ See [docs/runtime/database/07-wo-seg-migration.md](../docs/runtime/database/07-w ## `reference/writeonce-api/` and `reference/writeonce-app/` Earlier exploration snapshots that predate the v1 crates. Self-contained; no active build wiring. + +## `reference/linux/` and `reference/go/` (symlinks) + +Research source trees, gitignored. User-specific absolute paths — each contributor sets their own: + +```bash +ln -s reference/linux +ln -s reference/go +``` + +- **`reference/linux/`** — the Linux kernel source. Read `io_uring/`, `fs/notify/inotify/`, `kernel/eventfd.c`, `include/uapi/linux/*.h` when designing kernel-primitive modules. See per-primitive reference cards under [`docs/plan/linux/`](../docs/plan/linux/). +- **`reference/go/`** — the Go source tree. Read `src/runtime/netpoll_epoll.go`, `src/runtime/netpoll.go`, `src/runtime/asm_*.s`, `src/runtime/sys_linux_*.s` when designing the runtime layer — writeonce's `crates/rt/src/runtime/` mirrors Go's `src/runtime/` file-per-flavour naming. See [`docs/plan/assembly/`](../docs/plan/assembly/) for the asm policy doc that cites this tree.