shoney/writeonce
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

docs: status board on the compile-and-run milestone + CODE-LOGIC beside the code

Browse source 
- docs/00-status.md: NEXT PLAN is now iteration 5's strictness half (?T forced
  handling, pub(read) writes, using, #if) plus an ASan run over the workload;
  iterations 6 and 7 marked landed; a "Landed 2026-08-14" section records what
  actually shipped, and a new known-gaps block records what did not — lenient
  optionals, unenforced pub(read), the borrowed-Text-into-container hazard,
  json's Bool/float limits, net fd lifetime, no ASan over the workload, and no
  corpus fixtures for the new surface (by direction: the sample is the test)
- runtime/src/CODE-LOGIC.md: file map, the loader-is-the-only-validator and
  traps-never-leak invariants, the catch stack, records the VM fills but cannot
  name, class metadata + json, program mode, and where to look when it breaks
- compiler/src/CODE-LOGIC.md: the pipeline, why there are two type derivers and
  the stay-silent-when-underivable rule, how contextual values get a
  destination, the node-id/label contract between owner.ml and emit.ml, the
  four kinds of qualified call, predeclared records, the constant-interning
  trap, register discipline, and how to verify a change

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
shoney.arickathil 2026-08-14 17:20:59 +02:00
parent 940ad99c40
commit f95e9e4f63
3 changed files with 335 additions and 19 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

122
compiler/src/CODE-LOGIC.md Normal file
View file

@ -0,0 +1,122 @@
# `compiler/src` — how `woc` is put together
Written 2026-08-14, when the front end grew the language surface that compiles
`docs/examples/log-watcher`. The normative contracts it emits against are
[`docs/plan/oop-vm/00-wob-format.md`](../../docs/plan/oop-vm/00-wob-format.md)
and [`08-builtin-surface.md`](../../docs/plan/oop-vm/08-builtin-surface.md);
the diagnostic codes are catalogued in
[`01-error-catalog.md`](../../docs/plan/oop-vm/01-error-catalog.md).
## The pipeline
```
lexer.ml → parser.ml → types.ml → owner.ml → emit.ml → .wob
tokens AST symbols + move/drop/rc bytecode
typecheck tables
```
`bin/main.ml` drives it: discover files (a directory is one program), parse each,
collect declarations per file, check module edges, merge symbols, typecheck,
run the owner pass per file, then emit one image from every unit. `diag.ml`
accumulates every stage's diagnostics and sorts them by (file, line, col), so
ordering never depends on discovery order. `dump.ml` renders the stable text
dumps the golden tests diff; `disasm.ml` reads an image back.
Four things are worth knowing before editing any of it.
### 1. Two type derivers, deliberately
`types.ml`'s `confident_typ` and `emit.ml`'s `ty_of_expr` both answer "what type
is this expression?", in different languages (`Types.typ` vs `Ast.field_ty`) and
for different purposes: the first gates diagnostics, the second picks
instructions (EQ vs EQS, a container's element kinds, whether a value is owned).
They are kept in sync by hand, and both follow one rule: **stay silent when
underivable**. `confident_typ` returns `None`; the emitter falls back to `Int`.
That is why a check built on `typecheck_expr`'s `.typ` (which reports `Int` for
anything unresolved) produces false positives, and every new check should read
`confident_typ` instead.
A third table pair follows the same discipline: `Types.builtin_confident_ret`
and `emit.ml`'s `builtin_ret` give each builtin's return type. An omission there
is not a lost type — it is a **leak**, because the owner pass classifies a
binding as owned from exactly that answer.
### 2. Contextual values need a destination
`[]`, `[a, b]`, `{}` and `nil` have no type of their own. They take it from,
in order: a written `let` annotation, the field/parameter they are built into,
the enclosing method's declared return type (`fstate.f_ret`), or — for a
non-empty list — their own first element. With none of those, emission is a
diagnostic, never guessed bytecode: a container's element kinds *are* its
runtime drop plan, so a wrong guess leaks or double-frees. `nil` is the zero
word for every `?T` (the format doc's own rule), which is also why a comparison
against `nil` must lower to `EQ` and never `EQS`.
### 3. The owner pass hands the emitter tables, not decisions
`owner.ml` computes moves, scope-end drops, branch-join drops, rc sites and
residual borrow guards, keyed by **node id and label**. `emit.ml` looks them up
by the same keys. When a construct has arms — `switch`, `if`, `try` — both files
must agree on the label strings and on the arm ORDER (`switch_lowering_order`
moves `default` last in both). A silent mismatch means a drop that never runs.
`try`'s shape: the catch arm is an alternate flow joining the try arm, so
`analyze_try` snapshots the entry state, walks the body, restores, walks the
handler with `e` declared as an owned local, and then makes each arm drop what
the other moved. The handler starts from the *entry* state on purpose — a trap
can be raised after any prefix of the body, and claiming the body's moves
happened would drop values the VM already released.
### 4. Statics, modules and the stdlib all arrive as `Ident.member` calls
A qualified call's head can be four things, resolved in this order: a value with
a type (an ordinary method call), a class with a static method
(`Flock.held(x)` — `static_method`), a reserved stdlib module
(`fs.stat(path)` — `Types.stdlib_members`), or a `use` alias for a project
module. Adding a fifth kind means extending that chain in both `emit_call` and
`ty_of_expr`, and `confident_typ` for the diagnostic side.
The stdlib table is data: module, member, source arity, builtin id, return
type, and the predeclared record whose class id gets appended as the call's last
argument. `json.encode`/`json.decode` are the two exceptions with bespoke
lowering — encode needs its argument's static kind, and decode has no type at
all until an `as` names one, which is why `json.decode(t) as T` is one
instruction and a bare `json.decode(t)` is an error.
## Predeclared records
`Error` (a catch arm's error), `Stat`, `TimeParts`, `Proc` (stdlib results) are
declared by `types.ml`, not by any source file. They join the **merged** symbol
table only — one copy per file would read as a cross-file duplicate — and they
enter the class table only when a program actually needs one, so images that
predate the surface keep their exact class tables. Their field ORDER is the
contract with the runtime, which writes those fields by index.
## Emitting the class table (a trap to remember)
Field-name constants must be interned **with every other constant**, before the
constant pool is serialized. Interning during class-table serialization appends
constants the pool has already been written past: the image then references
constants it does not contain, and the loader rejects every class. That bug cost
a debugging round; the interning now happens beside `class_name_k`.
## Register discipline in `emit.ml`
Locals live below `f_nlocals`, temporaries from `f_temp` upward, and a
statement resets `f_temp` to `f_nlocals`. Any construct that writes into a `dst`
which might itself be a temp (`switch`, `try`, a ctor, a container literal) must
reserve `dst` before allocating more temps, or an arm-local `let` can be handed
the same register and clobber a live value before its drop runs. `emit_switch`
carries the comment explaining the ASan-confirmed leak that taught this.
## Verifying a change
- `just woc-test` — unit assertions plus the golden suite (token/AST/owner/bc
dumps and an OCaml re-implementation of the loader's validation). `WOC_BLESS=1`
regenerates goldens; read the diff before blessing, it is a contract change.
- `just oop-e2e` — the conformance corpus: `run/` byte-exact stdout,
`compile-fail/` exact diagnostic code, `trap/` exact trap code, `gc/` exact
collector trace, plus the single-binary smoke.
- `./compiler/_build/default/bin/woc --emit docs/examples/log-watcher -o /tmp/lw.wob`
— the acceptance workload. It must compile with zero diagnostics, and
`runtime/wovm /tmp/lw.wob watch <file> 2 1` must tail a live file and alert.

113
docs/00-status.md
View file

@ -19,20 +19,30 @@ Statuses: ✅ **done** · 🔄 **in progress** · ⬜ **pending** · ⏸ **hold*
## ▶ NEXT PLAN
**Story iteration 5 — language surface (Haxe-parity adoptions).**
**Close the gaps the log-watcher milestone left open.** The acceptance target
of the whole language track — _compile and run log-watcher_ — is **met** as of
2026-08-14: `docs/examples/log-watcher` (1285 lines, 7 files) compiles with
zero diagnostics, and the image runs (`wovm lw.wob watch app.log 2 1` tails a
live file, classifies levels and fires `ALERT … last entry is error, quiet for
2s`). What remains is the *strictness* half of iteration 5 plus one runtime
gate, in this order:
1. **`?T` forced handling** (plan 8 Task 6's diagnostics half, `WO-E211`–`E213`
still dead). Optionals are currently **lenient**: `nil` is the zero word, a
`?T` is usable where `T` is expected, and nothing narrows. The
representation and the comparisons are right; the refusals are missing.
2. **`pub(read)` write enforcement** — parsed and recorded on the field; the
typechecker does not yet refuse a write from outside the declaring class.
3. **`using` extensions and `#if` build flags + reject-row diagnostics** (plan 8
Tasks 7–8's remainder). Nothing in the workload needs them, so they are the
tail of the plan, not a blocker.
4. **ASan over the workload** — the corpus is ASan-clean, but log-watcher's own
run has never been under the sanitizer, and iteration 4's `gc/held-cycle`
leak is still open (see the known-gaps section).
Plan: [`plan/compiler/2026-08-01-haxe-parity-language.md`](plan/compiler/2026-08-01-haxe-parity-language.md) ·
Story slice: [`docs/stories/language-runtime-database/05-language-surface.md`](stories/language-runtime-database/05-language-surface.md)
The language grows from milestone grammar to a daily-driver surface: every
**adopt** row of the systems-track verdict table (switch expressions, typedef
records, `?T` optionals, enum payloads, try/catch, statics, `using`, modules,
`is`, `pub(read)`, `#if`) lands with a golden + must-fail fixture pair; every
**reject** row refuses with a doctrine-citing diagnostic. **First task: `?T`
forced handling (plan 8 Task 6)** — plumbed since iteration 3 but unenforced
(`WO-E211`–`E213` dead), and the log-watcher port (iterations 6–7) uses
optionals throughout in place of the Haxe original's sentinel values, so
nothing else in this plan can land ahead of it.
Two tracks run in this repo. The critical path is the **language track**:
iterations 3 → 4 → 5 → 6 → 7, ending at _compile and run log-watcher_. The
Rust-runtime track is shipped-and-maintained, not advancing.
@ -52,9 +62,9 @@ that sequences its tasks. Read one, approve, then the next starts.
| 2 | [VM core (`wovm`)](stories/language-runtime-database/02-vm-core.md) | ✅ |
| 3 | [Compiler front (`woc`)](stories/language-runtime-database/03-compiler-front.md) | ✅ (known gaps below) |
| 4 | [Single binary end-to-end](stories/language-runtime-database/04-single-binary-e2e.md) | ✅ (known gaps below) |
| 5 | [Language surface](stories/language-runtime-database/05-language-surface.md) | 🔄 **next** |
| 6 | [Program mode + stdlib](stories/language-runtime-database/06-program-mode-stdlib.md) | ⬜ |
| 7 | [log-watcher proof](stories/language-runtime-database/07-logwatcher-proof.md) | ⬜ acceptance |
| 5 | [Language surface](stories/language-runtime-database/05-language-surface.md) | 🔄 grammar done, strictness open |
| 6 | [Program mode + stdlib](stories/language-runtime-database/06-program-mode-stdlib.md) | ✅ (the surface log-watcher uses) |
| 7 | [log-watcher proof](stories/language-runtime-database/07-logwatcher-proof.md) | ✅ compiles and runs |
| 7b | [Inferred GC + mark-sweep](stories/language-runtime-database/07b-inferred-gc-mark-sweep.md) | ⬜ closes iteration 4's gate |
| 8 | [Shard-actor runtime](stories/language-runtime-database/08-shard-actor-runtime.md) | ⬜ |
| 9 | [Database engine](stories/language-runtime-database/09-database-engine.md) | ⬜ |
@ -67,12 +77,46 @@ that sequences its tasks. Read one, approve, then the next starts.
## In progress
| Track | Item | Where |
| -------- | ---------------------------------------------------------------------- | ---------------------------------------------------------- |
| Language | Iteration 5 — Haxe-parity language surface, `?T` forced handling first | [plan 8](plan/compiler/2026-08-01-haxe-parity-language.md) |
| Track | Item | Where |
| -------- | --------------------------------------------------------------------------- | ---------------------------------------------------------- |
| Language | Iteration 5's strictness half — `?T` forced handling, `pub(read)` writes, `using`, `#if` | [plan 8](plan/compiler/2026-08-01-haxe-parity-language.md) |
Nothing else should be started until iteration 5 lands. Off-critical-path work
is parked by explicit scope directive (2026-08-08).
Off-critical-path work is parked by explicit scope directive (2026-08-08).
### Landed 2026-08-14 — the compile-and-run milestone
One session, driven end to end by compiling `docs/examples/log-watcher` and
watching its diagnostic count fall (481 → 0). In order:
- **let annotations, container literals, statics, `pub(read)`** — `let x: multi
Text = []`, `map<K, V>`, `?T`; `[]`/`[a, b]`/`{}` as expressions; `static
const`/`static fn` with `Cls.fn(...)` calls; a `;` ends a statement so
one-line guard bodies parse.
- **try/catch over the trap system** (plan 8 Task 5) — VM catch frames
(`TRY`/`ENDTRY`), unwind-to-handler with the try region's own values
released, `err_fill` for the `{code, line, method, msg}` record, expression
and block catch arms. Uncaught traps unchanged.
- **`nil` + 23 text/container builtins** — len, byte_at, print_err,
starts_with/ends_with, index_of/last_index_of, substr, trim, to_lower,
char_of, parse_int, split/split_ws, join, slice, pop/shift, sort, reverse,
remove, key_at/val_at, multi_set.
- **`for k, v in m`** over a map, and `m[i] = v` for a `multi`.
- **the systems stdlib's OS half** (`runtime/src/sysio.c`) — fs, time, env,
net, proc behind the reserved module names, with predeclared `Stat`,
`TimeParts` and `Proc` records and the new `WO_T_IO` trap.
- **json** (`runtime/src/json.c`) + **`.wob` v2** — per-field names, referenced
classes and element kinds in the class table, so encode/decode are one
metadata-driven implementation; `json.decode(t) as T` is the language's only
cast, yielding `?T`.
- **program mode** — `fn main(args: multi Text) -> Int`, argv delivered by the
runtime, return value as the exit code.
- **two safety fixes found by running it**: `+` on `Text` was lowering to ADD
on two heap pointers (now WO-E201 pointing at `..`; seven sites in the sample
were corrected), and `x == nil` was lowering to EQS, which dereferences the
zero word (now EQ).
Gates at the end of that session: corpus 71/0, `woc` runtest 565/0, every
`wovm` unit gate green in both dispatch flavors.
---
@ -122,6 +166,37 @@ is parked by explicit scope directive (2026-08-08).
to `set` is under-counted and the collector can free it while the map still
points at it. Nothing in the corpus exercises this yet. See
[`oop-vm/08-builtin-surface.md`](plan/oop-vm/08-builtin-surface.md).
**Known gaps carried out of the 2026-08-14 compile-and-run milestone** —
recorded, not silently owed:
- **Optionals are lenient.** `?T` has its representation (the zero word) and
its comparisons, but `WO-E211`–`E213` are still dead: a `?T` may be used
where `T` is required, and nothing narrows inside an `if x != nil` branch.
The workload leans on that leniency today.
- **`pub(read)` is parsed, not enforced.** The marker rides on the field
(`Ast.field.pub_read`); no check refuses a write from outside the declaring
class yet.
- **`using` extensions and `#if` build flags are absent**, and the reject rows
(`extends`/`cast`/`Dynamic`/…) still have no doctrine-citing diagnostics —
plan 8 Tasks 7–8's remainder.
- **A borrowed non-constant Text pushed into a container is a double-free
hazard** — `push(m, v)`'s open gap, now shared by list literals (`[a, b]`)
and documented in `owner.ml`'s own comment. The workload's literals are
string constants or borrowed params handed straight to a stdlib call, so
nothing reachable today hits it.
- **json's two documented limits**: a `Bool` field encodes as `0`/`1` (the
class-table kind byte does not distinguish it from an integer), and a JSON
number with a fraction or exponent decodes by truncation.
- **`net` fd lifetime is the program's problem.** `net.close` exists; the
sample's MCP server never calls it, so a long-running `mcp` session leaks
descriptors. That is the sample's bug to fix, not the runtime's.
- **The workload has never run under ASan**, and iteration 4's `gc/held-cycle`
leak (above) is still open. The corpus itself stays ASan-clean.
- **No corpus fixtures cover the new surface.** By explicit direction
(2026-08-14) the acceptance for this work is the log-watcher program itself,
not fixture pairs; `tests/corpus/` still gates every pre-existing behavior
(71 checks, 0 failures).
- **E201/E203 and seven other `WO-E2xx` codes remain declared but unemitted**
— see [`oop-vm/01-error-catalog.md`](plan/oop-vm/01-error-catalog.md).
- **CLOSED — milestone-1's ASan gate (`just oop-accept`) failing on

119
runtime/src/CODE-LOGIC.md Normal file
View file

@ -0,0 +1,119 @@
# `runtime/src` — how the VM is put together
Written 2026-08-14, when the runtime grew the systems stdlib and json. Read
this before changing a file here; the normative contracts are
[`docs/plan/oop-vm/00-wob-format.md`](../../docs/plan/oop-vm/00-wob-format.md)
(the `.wob` format, opcodes, builtin ids) and
[`08-builtin-surface.md`](../../docs/plan/oop-vm/08-builtin-surface.md) (what
each builtin means in source terms). `wob.h` is the machine-readable twin of
the first: constants there and prose there must never disagree.
## The files, in dependency order
| file | what it owns |
| --- | --- |
| `wob.h` | every format constant: header offsets, field kinds, opcodes, builtin ids, trap codes, the 16-byte object header, the class descriptor |
| `obj.h/.c` | the per-shard arena, object allocation, `wo_str` (header + length + inline bytes, no NUL) |
| `cont.h/.c` | `multi` and `map` as native classes: struct heads in the arena, backing arrays malloc'd, map lookup a linear scan over parallel key/value arrays |
| `gc.h/.c` | the kind-directed dispatcher (`wo_drop_kind`/`wo_drop_obj`), refcounting for `@gc`, and the budgeted Bacon–Rajan cycle collector |
| `borrow.h/.c` | the borrow word: shared counts and the exclusive sentinel |
| `loader.h/.c` | parse and **validate** an image; the validation contract in its header comment is exactly what the interpreter may then assume |
| `vm.h/.c` | the register interpreter: window-overlap calls, dual-flavor dispatch, traps, unwinding, catch frames |
| `builtin.h/.c` | the pure builtins: print, containers, text |
| `sysio.c` | the OS half: `fs`, `time`, `env`, `net`, `proc` |
| `json.c` | `json.encode` / `json.decode`, driven by class metadata |
| `main.c` | the CLI: find an image (argument or embedded trailer), build argv, call the entry, map its result to an exit code |
`builtin.c`'s `wo_builtin` is the single entry point the interpreter calls; it
forwards ids at or above `WO_B_SYS_FIRST` to `sysio.c` and the json pair to
`json.c`. Splitting by translation unit keeps the kernel-touching code and the
format-walking code out of the hot builtin switch.
## Two invariants worth stating plainly
**The loader is the only validator.** Everything the interpreter skips
checking — opcode ranges, register operands, jump targets, builtin arities,
window sizes, table ordering — is checked once at load. The exceptions are
deliberate and documented: `GETF`/`SETF` field indexes and receiver shapes stay
runtime checks, because registers are untyped (the spec's residual-check
doctrine). If you add an opcode or a builtin, its validation goes in
`loader.c`'s switch and its arity in `b_arity`, or the interpreter is running
unvalidated bytes.
**Traps never leak.** A trap unwinds frames innermost-outward, and in each one
the drop-table entry governing that frame's current instruction says which
registers hold owned or counted values. The governing instruction is the
trapping pc for the innermost frame and the CALL (saved pc − 1) for every outer
one. Registers are nulled as they are released, so window overlap cannot
double-free.
## try/catch (the catch stack)
`TRY A sBx` pushes `{depth, handler pc, error register}`; `ENDTRY` pops it. On a
trap with a catch frame live, `vm_trap`:
1. fills `vm->caught` (the same structured error the uncaught surface prints),
2. unwinds every frame **above** the catching one, exactly as an uncaught trap
would,
3. releases what the try region owned **in** the catching frame — the
difference between the drop entry at the trapping instruction and the entry
at the handler pc, which is why the compiler must record an entry at the
handler,
4. points that frame at the handler and returns 0, so `TRAPF` reloads and keeps
interpreting.
A frame that returns pops the catch frames it registered (`DROP_CATCHES`), so a
`return` out of a try region cannot leave a handler aimed at a dead window.
With `ncatch == 0` every trap behaves byte-for-byte as it did before the
feature existed — that is the property to preserve when touching this code.
## Records the VM fills but does not know
Three builtins return a *record*: `fs.stat`, `time.local`, `proc.run`, plus
`err_fill` for a catch arm. The VM cannot name a source type, so the compiler
passes the record's **class id** as the call's last argument and the builtin
fills fields by index. The field order is therefore a contract, written beside
each case in `sysio.c` and mirrored in `compiler/src/types.ml`'s predeclared
records. Change one side and the other silently writes to the wrong slot.
## Class metadata and json (`.wob` v2)
The class table carries, per field, its name constant, the class it refers to
(or a json-raw marker) and a container field's element kinds. That is what lets
`json.c` be one implementation for every shape instead of per-type generated
code:
- **encode** takes the top-level value's *static* kind from the compiler,
because a register alone cannot say whether it holds an i64 or a pointer.
Everything nested comes from object headers (which carry `class_id`) and the
class table.
- **decode** parses and binds straight into the target class: keys matched
against field names, a nested object built as that field's class, an array as
a `multi` of that field's element kind, unknown keys skipped, absent keys left
as the zero word (nil). Malformed input yields nil rather than trapping —
that is what makes `json.decode(t) as T` a checked decode.
Two limits are inherent to the kind byte and are documented, not bugs to
discover: a `Bool` field encodes as `0`/`1`, and a fractional JSON number
decodes by truncation.
## Program mode
`main.c` accepts an entry taking no arguments or exactly one `multi Text`. The
list holds the program's **own** arguments — not the program name, and not the
image path a `wovm image.wob args...` invocation carries — so `args[0]` is the
first real argument. The entry's return value is the process exit code (low
byte); a trap is exit 1 with the fixed `trap N in METHOD at line L: MESSAGE`
line on stderr, which the conformance harness parses.
## Where to look when something breaks
- A wild pointer inside a builtin usually means the *compiler* put the wrong
thing in a register: check the method's disassembly (`woc --dump-bc`) before
suspecting the C.
- `make -C runtime wovm-asan` builds the sanitized binary; the unit suites
(`just wovm-test`) run every `test/test_*.c` under ASan+UBSan in both
dispatch flavors, so a fallback-only bug cannot hide.
- `runtime/test/wob_build.c` is an independent image assembler. A
builder/loader disagreement shows up as a unit-test failure, which is the
point of having two encoders.