From 68879f27e2adc4d2cd6a77c723d8a29a01d9660a Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Fri, 14 Aug 2026 17:20:59 +0200 Subject: [PATCH] docs: status board on the compile-and-run milestone + CODE-LOGIC beside the code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs/00-status.md: NEXT PLAN is now iteration 5's strictness half (?T forced handling, pub(read) writes, using, #if) plus an ASan run over the workload; iterations 6 and 7 marked landed; a "Landed 2026-08-14" section records what actually shipped, and a new known-gaps block records what did not — lenient optionals, unenforced pub(read), the borrowed-Text-into-container hazard, json's Bool/float limits, net fd lifetime, no ASan over the workload, and no corpus fixtures for the new surface (by direction: the sample is the test) - runtime/src/CODE-LOGIC.md: file map, the loader-is-the-only-validator and traps-never-leak invariants, the catch stack, records the VM fills but cannot name, class metadata + json, program mode, and where to look when it breaks - compiler/src/CODE-LOGIC.md: the pipeline, why there are two type derivers and the stay-silent-when-underivable rule, how contextual values get a destination, the node-id/label contract between owner.ml and emit.ml, the four kinds of qualified call, predeclared records, the constant-interning trap, register discipline, and how to verify a change Co-Authored-By: Claude Opus 5 (1M context) --- compiler/src/CODE-LOGIC.md | 122 +++++++++++++++++++++++++++++++++++++ docs/00-status.md | 113 ++++++++++++++++++++++++++++------ runtime/src/CODE-LOGIC.md | 119 ++++++++++++++++++++++++++++++++++++ 3 files changed, 335 insertions(+), 19 deletions(-) create mode 100644 compiler/src/CODE-LOGIC.md create mode 100644 runtime/src/CODE-LOGIC.md diff --git a/compiler/src/CODE-LOGIC.md b/compiler/src/CODE-LOGIC.md new file mode 100644 index 0000000..b6c3f29 --- /dev/null +++ b/compiler/src/CODE-LOGIC.md @@ -0,0 +1,122 @@ +# `compiler/src` — how `woc` is put together + +Written 2026-08-14, when the front end grew the language surface that compiles +`docs/examples/log-watcher`. The normative contracts it emits against are +[`docs/plan/oop-vm/00-wob-format.md`](../../docs/plan/oop-vm/00-wob-format.md) +and [`08-builtin-surface.md`](../../docs/plan/oop-vm/08-builtin-surface.md); +the diagnostic codes are catalogued in +[`01-error-catalog.md`](../../docs/plan/oop-vm/01-error-catalog.md). + +## The pipeline + +``` +lexer.ml → parser.ml → types.ml → owner.ml → emit.ml → .wob +tokens AST symbols + move/drop/rc bytecode + typecheck tables +``` + +`bin/main.ml` drives it: discover files (a directory is one program), parse each, +collect declarations per file, check module edges, merge symbols, typecheck, +run the owner pass per file, then emit one image from every unit. `diag.ml` +accumulates every stage's diagnostics and sorts them by (file, line, col), so +ordering never depends on discovery order. `dump.ml` renders the stable text +dumps the golden tests diff; `disasm.ml` reads an image back. + +Four things are worth knowing before editing any of it. + +### 1. Two type derivers, deliberately + +`types.ml`'s `confident_typ` and `emit.ml`'s `ty_of_expr` both answer "what type +is this expression?", in different languages (`Types.typ` vs `Ast.field_ty`) and +for different purposes: the first gates diagnostics, the second picks +instructions (EQ vs EQS, a container's element kinds, whether a value is owned). +They are kept in sync by hand, and both follow one rule: **stay silent when +underivable**. `confident_typ` returns `None`; the emitter falls back to `Int`. +That is why a check built on `typecheck_expr`'s `.typ` (which reports `Int` for +anything unresolved) produces false positives, and every new check should read +`confident_typ` instead. + +A third table pair follows the same discipline: `Types.builtin_confident_ret` +and `emit.ml`'s `builtin_ret` give each builtin's return type. An omission there +is not a lost type — it is a **leak**, because the owner pass classifies a +binding as owned from exactly that answer. + +### 2. Contextual values need a destination + +`[]`, `[a, b]`, `{}` and `nil` have no type of their own. They take it from, +in order: a written `let` annotation, the field/parameter they are built into, +the enclosing method's declared return type (`fstate.f_ret`), or — for a +non-empty list — their own first element. With none of those, emission is a +diagnostic, never guessed bytecode: a container's element kinds *are* its +runtime drop plan, so a wrong guess leaks or double-frees. `nil` is the zero +word for every `?T` (the format doc's own rule), which is also why a comparison +against `nil` must lower to `EQ` and never `EQS`. + +### 3. The owner pass hands the emitter tables, not decisions + +`owner.ml` computes moves, scope-end drops, branch-join drops, rc sites and +residual borrow guards, keyed by **node id and label**. `emit.ml` looks them up +by the same keys. When a construct has arms — `switch`, `if`, `try` — both files +must agree on the label strings and on the arm ORDER (`switch_lowering_order` +moves `default` last in both). A silent mismatch means a drop that never runs. + +`try`'s shape: the catch arm is an alternate flow joining the try arm, so +`analyze_try` snapshots the entry state, walks the body, restores, walks the +handler with `e` declared as an owned local, and then makes each arm drop what +the other moved. The handler starts from the *entry* state on purpose — a trap +can be raised after any prefix of the body, and claiming the body's moves +happened would drop values the VM already released. + +### 4. Statics, modules and the stdlib all arrive as `Ident.member` calls + +A qualified call's head can be four things, resolved in this order: a value with +a type (an ordinary method call), a class with a static method +(`Flock.held(x)` — `static_method`), a reserved stdlib module +(`fs.stat(path)` — `Types.stdlib_members`), or a `use` alias for a project +module. Adding a fifth kind means extending that chain in both `emit_call` and +`ty_of_expr`, and `confident_typ` for the diagnostic side. + +The stdlib table is data: module, member, source arity, builtin id, return +type, and the predeclared record whose class id gets appended as the call's last +argument. `json.encode`/`json.decode` are the two exceptions with bespoke +lowering — encode needs its argument's static kind, and decode has no type at +all until an `as` names one, which is why `json.decode(t) as T` is one +instruction and a bare `json.decode(t)` is an error. + +## Predeclared records + +`Error` (a catch arm's error), `Stat`, `TimeParts`, `Proc` (stdlib results) are +declared by `types.ml`, not by any source file. They join the **merged** symbol +table only — one copy per file would read as a cross-file duplicate — and they +enter the class table only when a program actually needs one, so images that +predate the surface keep their exact class tables. Their field ORDER is the +contract with the runtime, which writes those fields by index. + +## Emitting the class table (a trap to remember) + +Field-name constants must be interned **with every other constant**, before the +constant pool is serialized. Interning during class-table serialization appends +constants the pool has already been written past: the image then references +constants it does not contain, and the loader rejects every class. That bug cost +a debugging round; the interning now happens beside `class_name_k`. + +## Register discipline in `emit.ml` + +Locals live below `f_nlocals`, temporaries from `f_temp` upward, and a +statement resets `f_temp` to `f_nlocals`. Any construct that writes into a `dst` +which might itself be a temp (`switch`, `try`, a ctor, a container literal) must +reserve `dst` before allocating more temps, or an arm-local `let` can be handed +the same register and clobber a live value before its drop runs. `emit_switch` +carries the comment explaining the ASan-confirmed leak that taught this. + +## Verifying a change + +- `just woc-test` — unit assertions plus the golden suite (token/AST/owner/bc + dumps and an OCaml re-implementation of the loader's validation). `WOC_BLESS=1` + regenerates goldens; read the diff before blessing, it is a contract change. +- `just oop-e2e` — the conformance corpus: `run/` byte-exact stdout, + `compile-fail/` exact diagnostic code, `trap/` exact trap code, `gc/` exact + collector trace, plus the single-binary smoke. +- `./compiler/_build/default/bin/woc --emit docs/examples/log-watcher -o /tmp/lw.wob` + — the acceptance workload. It must compile with zero diagnostics, and + `runtime/wovm /tmp/lw.wob watch 2 1` must tail a live file and alert. diff --git a/docs/00-status.md b/docs/00-status.md index 03e39e4..d08c21c 100644 --- a/docs/00-status.md +++ b/docs/00-status.md @@ -19,20 +19,30 @@ Statuses: ✅ **done** · 🔄 **in progress** · ⬜ **pending** · ⏸ **hold* ## ▶ NEXT PLAN -**Story iteration 5 — language surface (Haxe-parity adoptions).** +**Close the gaps the log-watcher milestone left open.** The acceptance target +of the whole language track — _compile and run log-watcher_ — is **met** as of +2026-08-14: `docs/examples/log-watcher` (1285 lines, 7 files) compiles with +zero diagnostics, and the image runs (`wovm lw.wob watch app.log 2 1` tails a +live file, classifies levels and fires `ALERT … last entry is error, quiet for +2s`). What remains is the *strictness* half of iteration 5 plus one runtime +gate, in this order: + +1. **`?T` forced handling** (plan 8 Task 6's diagnostics half, `WO-E211`–`E213` + still dead). Optionals are currently **lenient**: `nil` is the zero word, a + `?T` is usable where `T` is expected, and nothing narrows. The + representation and the comparisons are right; the refusals are missing. +2. **`pub(read)` write enforcement** — parsed and recorded on the field; the + typechecker does not yet refuse a write from outside the declaring class. +3. **`using` extensions and `#if` build flags + reject-row diagnostics** (plan 8 + Tasks 7–8's remainder). Nothing in the workload needs them, so they are the + tail of the plan, not a blocker. +4. **ASan over the workload** — the corpus is ASan-clean, but log-watcher's own + run has never been under the sanitizer, and iteration 4's `gc/held-cycle` + leak is still open (see the known-gaps section). + Plan: [`plan/compiler/2026-08-01-haxe-parity-language.md`](plan/compiler/2026-08-01-haxe-parity-language.md) · Story slice: [`docs/stories/language-runtime-database/05-language-surface.md`](stories/language-runtime-database/05-language-surface.md) -The language grows from milestone grammar to a daily-driver surface: every -**adopt** row of the systems-track verdict table (switch expressions, typedef -records, `?T` optionals, enum payloads, try/catch, statics, `using`, modules, -`is`, `pub(read)`, `#if`) lands with a golden + must-fail fixture pair; every -**reject** row refuses with a doctrine-citing diagnostic. **First task: `?T` -forced handling (plan 8 Task 6)** — plumbed since iteration 3 but unenforced -(`WO-E211`–`E213` dead), and the log-watcher port (iterations 6–7) uses -optionals throughout in place of the Haxe original's sentinel values, so -nothing else in this plan can land ahead of it. - Two tracks run in this repo. The critical path is the **language track**: iterations 3 → 4 → 5 → 6 → 7, ending at _compile and run log-watcher_. The Rust-runtime track is shipped-and-maintained, not advancing. @@ -52,9 +62,9 @@ that sequences its tasks. Read one, approve, then the next starts. | 2 | [VM core (`wovm`)](stories/language-runtime-database/02-vm-core.md) | ✅ | | 3 | [Compiler front (`woc`)](stories/language-runtime-database/03-compiler-front.md) | ✅ (known gaps below) | | 4 | [Single binary end-to-end](stories/language-runtime-database/04-single-binary-e2e.md) | ✅ (known gaps below) | -| 5 | [Language surface](stories/language-runtime-database/05-language-surface.md) | 🔄 **next** | -| 6 | [Program mode + stdlib](stories/language-runtime-database/06-program-mode-stdlib.md) | ⬜ | -| 7 | [log-watcher proof](stories/language-runtime-database/07-logwatcher-proof.md) | ⬜ acceptance | +| 5 | [Language surface](stories/language-runtime-database/05-language-surface.md) | 🔄 grammar done, strictness open | +| 6 | [Program mode + stdlib](stories/language-runtime-database/06-program-mode-stdlib.md) | ✅ (the surface log-watcher uses) | +| 7 | [log-watcher proof](stories/language-runtime-database/07-logwatcher-proof.md) | ✅ compiles and runs | | 7b | [Inferred GC + mark-sweep](stories/language-runtime-database/07b-inferred-gc-mark-sweep.md) | ⬜ closes iteration 4's gate | | 8 | [Shard-actor runtime](stories/language-runtime-database/08-shard-actor-runtime.md) | ⬜ | | 9 | [Database engine](stories/language-runtime-database/09-database-engine.md) | ⬜ | @@ -67,12 +77,46 @@ that sequences its tasks. Read one, approve, then the next starts. ## In progress -| Track | Item | Where | -| -------- | ---------------------------------------------------------------------- | ---------------------------------------------------------- | -| Language | Iteration 5 — Haxe-parity language surface, `?T` forced handling first | [plan 8](plan/compiler/2026-08-01-haxe-parity-language.md) | +| Track | Item | Where | +| -------- | --------------------------------------------------------------------------- | ---------------------------------------------------------- | +| Language | Iteration 5's strictness half — `?T` forced handling, `pub(read)` writes, `using`, `#if` | [plan 8](plan/compiler/2026-08-01-haxe-parity-language.md) | -Nothing else should be started until iteration 5 lands. Off-critical-path work -is parked by explicit scope directive (2026-08-08). +Off-critical-path work is parked by explicit scope directive (2026-08-08). + +### Landed 2026-08-14 — the compile-and-run milestone + +One session, driven end to end by compiling `docs/examples/log-watcher` and +watching its diagnostic count fall (481 → 0). In order: + +- **let annotations, container literals, statics, `pub(read)`** — `let x: multi + Text = []`, `map`, `?T`; `[]`/`[a, b]`/`{}` as expressions; `static + const`/`static fn` with `Cls.fn(...)` calls; a `;` ends a statement so + one-line guard bodies parse. +- **try/catch over the trap system** (plan 8 Task 5) — VM catch frames + (`TRY`/`ENDTRY`), unwind-to-handler with the try region's own values + released, `err_fill` for the `{code, line, method, msg}` record, expression + and block catch arms. Uncaught traps unchanged. +- **`nil` + 23 text/container builtins** — len, byte_at, print_err, + starts_with/ends_with, index_of/last_index_of, substr, trim, to_lower, + char_of, parse_int, split/split_ws, join, slice, pop/shift, sort, reverse, + remove, key_at/val_at, multi_set. +- **`for k, v in m`** over a map, and `m[i] = v` for a `multi`. +- **the systems stdlib's OS half** (`runtime/src/sysio.c`) — fs, time, env, + net, proc behind the reserved module names, with predeclared `Stat`, + `TimeParts` and `Proc` records and the new `WO_T_IO` trap. +- **json** (`runtime/src/json.c`) + **`.wob` v2** — per-field names, referenced + classes and element kinds in the class table, so encode/decode are one + metadata-driven implementation; `json.decode(t) as T` is the language's only + cast, yielding `?T`. +- **program mode** — `fn main(args: multi Text) -> Int`, argv delivered by the + runtime, return value as the exit code. +- **two safety fixes found by running it**: `+` on `Text` was lowering to ADD + on two heap pointers (now WO-E201 pointing at `..`; seven sites in the sample + were corrected), and `x == nil` was lowering to EQS, which dereferences the + zero word (now EQ). + +Gates at the end of that session: corpus 71/0, `woc` runtest 565/0, every +`wovm` unit gate green in both dispatch flavors. --- @@ -122,6 +166,37 @@ is parked by explicit scope directive (2026-08-08). to `set` is under-counted and the collector can free it while the map still points at it. Nothing in the corpus exercises this yet. See [`oop-vm/08-builtin-surface.md`](plan/oop-vm/08-builtin-surface.md). + +**Known gaps carried out of the 2026-08-14 compile-and-run milestone** — +recorded, not silently owed: + +- **Optionals are lenient.** `?T` has its representation (the zero word) and + its comparisons, but `WO-E211`–`E213` are still dead: a `?T` may be used + where `T` is required, and nothing narrows inside an `if x != nil` branch. + The workload leans on that leniency today. +- **`pub(read)` is parsed, not enforced.** The marker rides on the field + (`Ast.field.pub_read`); no check refuses a write from outside the declaring + class yet. +- **`using` extensions and `#if` build flags are absent**, and the reject rows + (`extends`/`cast`/`Dynamic`/…) still have no doctrine-citing diagnostics — + plan 8 Tasks 7–8's remainder. +- **A borrowed non-constant Text pushed into a container is a double-free + hazard** — `push(m, v)`'s open gap, now shared by list literals (`[a, b]`) + and documented in `owner.ml`'s own comment. The workload's literals are + string constants or borrowed params handed straight to a stdlib call, so + nothing reachable today hits it. +- **json's two documented limits**: a `Bool` field encodes as `0`/`1` (the + class-table kind byte does not distinguish it from an integer), and a JSON + number with a fraction or exponent decodes by truncation. +- **`net` fd lifetime is the program's problem.** `net.close` exists; the + sample's MCP server never calls it, so a long-running `mcp` session leaks + descriptors. That is the sample's bug to fix, not the runtime's. +- **The workload has never run under ASan**, and iteration 4's `gc/held-cycle` + leak (above) is still open. The corpus itself stays ASan-clean. +- **No corpus fixtures cover the new surface.** By explicit direction + (2026-08-14) the acceptance for this work is the log-watcher program itself, + not fixture pairs; `tests/corpus/` still gates every pre-existing behavior + (71 checks, 0 failures). - **E201/E203 and seven other `WO-E2xx` codes remain declared but unemitted** — see [`oop-vm/01-error-catalog.md`](plan/oop-vm/01-error-catalog.md). - **CLOSED — milestone-1's ASan gate (`just oop-accept`) failing on diff --git a/runtime/src/CODE-LOGIC.md b/runtime/src/CODE-LOGIC.md new file mode 100644 index 0000000..07984ab --- /dev/null +++ b/runtime/src/CODE-LOGIC.md @@ -0,0 +1,119 @@ +# `runtime/src` — how the VM is put together + +Written 2026-08-14, when the runtime grew the systems stdlib and json. Read +this before changing a file here; the normative contracts are +[`docs/plan/oop-vm/00-wob-format.md`](../../docs/plan/oop-vm/00-wob-format.md) +(the `.wob` format, opcodes, builtin ids) and +[`08-builtin-surface.md`](../../docs/plan/oop-vm/08-builtin-surface.md) (what +each builtin means in source terms). `wob.h` is the machine-readable twin of +the first: constants there and prose there must never disagree. + +## The files, in dependency order + +| file | what it owns | +| --- | --- | +| `wob.h` | every format constant: header offsets, field kinds, opcodes, builtin ids, trap codes, the 16-byte object header, the class descriptor | +| `obj.h/.c` | the per-shard arena, object allocation, `wo_str` (header + length + inline bytes, no NUL) | +| `cont.h/.c` | `multi` and `map` as native classes: struct heads in the arena, backing arrays malloc'd, map lookup a linear scan over parallel key/value arrays | +| `gc.h/.c` | the kind-directed dispatcher (`wo_drop_kind`/`wo_drop_obj`), refcounting for `@gc`, and the budgeted Bacon–Rajan cycle collector | +| `borrow.h/.c` | the borrow word: shared counts and the exclusive sentinel | +| `loader.h/.c` | parse and **validate** an image; the validation contract in its header comment is exactly what the interpreter may then assume | +| `vm.h/.c` | the register interpreter: window-overlap calls, dual-flavor dispatch, traps, unwinding, catch frames | +| `builtin.h/.c` | the pure builtins: print, containers, text | +| `sysio.c` | the OS half: `fs`, `time`, `env`, `net`, `proc` | +| `json.c` | `json.encode` / `json.decode`, driven by class metadata | +| `main.c` | the CLI: find an image (argument or embedded trailer), build argv, call the entry, map its result to an exit code | + +`builtin.c`'s `wo_builtin` is the single entry point the interpreter calls; it +forwards ids at or above `WO_B_SYS_FIRST` to `sysio.c` and the json pair to +`json.c`. Splitting by translation unit keeps the kernel-touching code and the +format-walking code out of the hot builtin switch. + +## Two invariants worth stating plainly + +**The loader is the only validator.** Everything the interpreter skips +checking — opcode ranges, register operands, jump targets, builtin arities, +window sizes, table ordering — is checked once at load. The exceptions are +deliberate and documented: `GETF`/`SETF` field indexes and receiver shapes stay +runtime checks, because registers are untyped (the spec's residual-check +doctrine). If you add an opcode or a builtin, its validation goes in +`loader.c`'s switch and its arity in `b_arity`, or the interpreter is running +unvalidated bytes. + +**Traps never leak.** A trap unwinds frames innermost-outward, and in each one +the drop-table entry governing that frame's current instruction says which +registers hold owned or counted values. The governing instruction is the +trapping pc for the innermost frame and the CALL (saved pc − 1) for every outer +one. Registers are nulled as they are released, so window overlap cannot +double-free. + +## try/catch (the catch stack) + +`TRY A sBx` pushes `{depth, handler pc, error register}`; `ENDTRY` pops it. On a +trap with a catch frame live, `vm_trap`: + +1. fills `vm->caught` (the same structured error the uncaught surface prints), +2. unwinds every frame **above** the catching one, exactly as an uncaught trap + would, +3. releases what the try region owned **in** the catching frame — the + difference between the drop entry at the trapping instruction and the entry + at the handler pc, which is why the compiler must record an entry at the + handler, +4. points that frame at the handler and returns 0, so `TRAPF` reloads and keeps + interpreting. + +A frame that returns pops the catch frames it registered (`DROP_CATCHES`), so a +`return` out of a try region cannot leave a handler aimed at a dead window. +With `ncatch == 0` every trap behaves byte-for-byte as it did before the +feature existed — that is the property to preserve when touching this code. + +## Records the VM fills but does not know + +Three builtins return a *record*: `fs.stat`, `time.local`, `proc.run`, plus +`err_fill` for a catch arm. The VM cannot name a source type, so the compiler +passes the record's **class id** as the call's last argument and the builtin +fills fields by index. The field order is therefore a contract, written beside +each case in `sysio.c` and mirrored in `compiler/src/types.ml`'s predeclared +records. Change one side and the other silently writes to the wrong slot. + +## Class metadata and json (`.wob` v2) + +The class table carries, per field, its name constant, the class it refers to +(or a json-raw marker) and a container field's element kinds. That is what lets +`json.c` be one implementation for every shape instead of per-type generated +code: + +- **encode** takes the top-level value's *static* kind from the compiler, + because a register alone cannot say whether it holds an i64 or a pointer. + Everything nested comes from object headers (which carry `class_id`) and the + class table. +- **decode** parses and binds straight into the target class: keys matched + against field names, a nested object built as that field's class, an array as + a `multi` of that field's element kind, unknown keys skipped, absent keys left + as the zero word (nil). Malformed input yields nil rather than trapping — + that is what makes `json.decode(t) as T` a checked decode. + +Two limits are inherent to the kind byte and are documented, not bugs to +discover: a `Bool` field encodes as `0`/`1`, and a fractional JSON number +decodes by truncation. + +## Program mode + +`main.c` accepts an entry taking no arguments or exactly one `multi Text`. The +list holds the program's **own** arguments — not the program name, and not the +image path a `wovm image.wob args...` invocation carries — so `args[0]` is the +first real argument. The entry's return value is the process exit code (low +byte); a trap is exit 1 with the fixed `trap N in METHOD at line L: MESSAGE` +line on stderr, which the conformance harness parses. + +## Where to look when something breaks + +- A wild pointer inside a builtin usually means the *compiler* put the wrong + thing in a register: check the method's disassembly (`woc --dump-bc`) before + suspecting the C. +- `make -C runtime wovm-asan` builds the sanitized binary; the unit suites + (`just wovm-test`) run every `test/test_*.c` under ASan+UBSan in both + dispatch flavors, so a fallback-only bug cannot hide. +- `runtime/test/wob_build.c` is an independent image assembler. A + builder/loader disagreement shows up as a unit-test failure, which is the + point of having two encoders.