diff --git a/.gitignore b/.gitignore index 57979a5..bf8a5ea 100644 --- a/.gitignore +++ b/.gitignore @@ -69,4 +69,9 @@ docs/examples/agent-loop/ docs/examples/mcp-think/ docs/plan/exploration/ docs/plan/oop-vm/ +# docs/plan/oop-vm/ carries the compiler<->VM normative contracts +# (.wob format, error catalog) both plan tracks cite -- carve it back +# out of the blanket docs/plan/ ignore above; the other rules here are +# untouched. +!docs/plan/oop-vm/ docs/superpowers/plans/ diff --git a/compiler/README.md b/compiler/README.md index a79a354..9c97af5 100644 --- a/compiler/README.md +++ b/compiler/README.md @@ -1,8 +1,8 @@ # compiler/ — the OCaml `woc` compiler -Lexer → parser → typechecker → ownership pass → bytecode emitter for `.wo`. OCaml stdlib only (no Menhir, no ppx); dune is the build runner. Sibling of the C `wovm` bytecode VM (`runtime/`) — the two halves of the OOP track's spec (`docs/superpowers/specs/2026-08-01-oop-compiler-vm-design.md`) meet at plan 3 (`.wob` emission). +Lexer → parser → typechecker → ownership pass, for `.wo`. OCaml stdlib only (no Menhir, no ppx); dune is the build runner. Sibling of the C `wovm` bytecode VM (`runtime/`) — the two halves of the OOP track's spec (`docs/superpowers/specs/2026-08-01-oop-compiler-vm-design.md`) meet at plan 3 (`.wob` emission; not built yet). -**Stage: Task 1 scaffold.** `src/` is currently an empty library and `bin/main.ml` is a CLI stub that only wires the exit-code contract below — no lexing, parsing, or checking happens yet. Tasks 2–8 (`compiler/plan/2026-08-01-woc-compiler-front.md`) fill in diagnostics, lexer, parser, typechecker, and the ownership pass in that order. +**Stage: plan 2 (`docs/plan/compiler/2026-08-01-woc-compiler-front.md`) complete, Tasks 1–8.** Diagnostics, lexer, parser (declarations + statements/expressions), typechecker (symbols, field kinds, structural interfaces, `?T` nullable), and the MVS ownership pass are all implemented and wired into the `woc` executable. Bytecode emission and `.wob` output are plan 3 — not started. ## Requirements @@ -12,19 +12,31 @@ OCaml 4.14.1, dune 3.14.0 — Ubuntu 24.04 apt packages (`sudo apt install ocaml ```bash cd compiler && dune build # -> _build/default/bin/woc -cd compiler && dune runtest # golden suite (empty until Task 3 adds test/dune) +cd compiler && dune runtest # test_diag unit checks + runner golden/CLI-smoke suite just woc-build # same, from the repo root just woc-test # same, from the repo root ``` -`woc` with no arguments (or the wrong number of arguments) prints usage to stderr and exits 2; given one path argument it exits 0 if the path exists, 2 if it doesn't. Exit-code contract, established now and enforced fully once diagnostics land in Task 2: **0** clean compile, **1** diagnostics reported, **2** usage/IO failure. +`WOC_BLESS=1 dune runtest` (from `compiler/`) rewrites golden `.expected` files to match current output — use it once, by hand, to seed or intentionally update a fixture. + +## Running `woc` + +``` +woc # compile (lex, parse, typecheck, ownership-check); nothing prints on success +woc --dump-tokens # stdout: one line per lexed token +woc --dump-ast # stdout: the declaration + body AST, indented +woc --dump-owner # stdout: the ownership pass's four tables (moves, drops, rc, residual) +``` + +`` is a single `.wo` file or a directory. A directory is discovered recursively for every `.wo` file under it — same contract as `wo run` (`crates/rt/src/lib.rs::discover`): dot-prefixed entries and `target`/`data`/`node_modules` are skipped, results are sorted by path. Every discovered file compiles as one program (declarations in one file resolve for bodies in another, regardless of discovery order); diagnostics from every file and every stage print sorted by `(file, line, col)`. For multi-file `--dump-*` output, each file's dump is preceded by a `=== path ===` header line (`compiler/src/dump.ml`'s `file_header`) — a single-file run never prints one. + +Diagnostics render as `file:line:col: severity CODE: message` plus a source excerpt with a caret; every shipped code is cataloged in `docs/plan/oop-vm/01-error-catalog.md`. Exit codes: **0** clean compile, **1** diagnostics reported, **2** usage/IO failure. ## Layout -- `src/` — one module per stage, added as each task lands: diag, token, lexer, ast, parser, types, owner, dump -- `bin/` — the `woc` executable (check / emit / build modes land in later tasks) -- `test/` — golden runner; `test/golden/` holds fixtures per stage (Task 3 onward) -- `plan/` — compiler-track docs: [`architecture.md`](plan/architecture.md) (pipeline, module contracts, reference-study map) + plans 2, 3, 8 +- `src/` — one module per stage: `diag` (diagnostics, collector, exit-code decision), `token`/`lexer`, `ast`/`parser`, `types` (typechecker), `owner` (MVS ownership pass), `dump` (stable text dumps for all of the above) +- `bin/` — the `woc` executable: CLI parsing, file discovery, the multi-file/cross-file driver +- `test/` — `runner.ml` (golden runner + CLI smoke) and `test_diag.ml` (diag.ml unit checks); `test/golden//` holds one-file-per-fixture goldens (`tokens`, `ast`, `owner`, `owner-err`); `test/fixtures/driver/` holds the multi-file CLI-smoke fixtures (directory discovery, cross-file symbols, diagnostic ordering) that don't fit the one-`.wo`-file-per-fixture golden shape -Governing docs: spec `docs/superpowers/specs/2026-08-01-oop-compiler-vm-design.md`; plans 2, 3, 8 in `compiler/plan/`. Format contract: `docs/plan/oop-vm/00-wob-format.md`. +Governing docs (all under `docs/`, not here — this file stays an orientation README): spec `docs/superpowers/specs/2026-08-01-oop-compiler-vm-design.md`; plan `docs/plan/compiler/2026-08-01-woc-compiler-front.md` (+ `architecture.md`, `nullable-types-implementation.md`, `2026-08-01-haxe-parity-language.md`, `2026-08-01-wob-emit-e2e-single-binary.md` in the same directory). Format contract: `docs/plan/oop-vm/00-wob-format.md`. Error catalog: `docs/plan/oop-vm/01-error-catalog.md`. diff --git a/compiler/bin/main.ml b/compiler/bin/main.ml index 8a17e9d..20963eb 100644 --- a/compiler/bin/main.ml +++ b/compiler/bin/main.ml @@ -7,30 +7,53 @@ (compiler/src/{ast,parser}.ml) behind --dump-ast, printing a stable, golden-diffed AST dump (compiler/src/dump.ml) to stdout. Tasks 5-7 add statement/expression parsing, the typechecker, and the ownership - pass behind their own --dump-* flags; Task 8 adds directory - discovery and multi-file programs, at which point the bare - `woc ` form below starts actually compiling instead of just - checking existence. + pass behind their own --dump-* flags. Task 8 adds directory + discovery and multi-file programs: may now be a single .wo + file or a directory, recursively discovered the same way `wo run` + discovers a project (compiler/bin/main.ml's discover_dir mirrors + crates/rt/src/lib.rs::discover — skips dot-prefixed entries and + target/data/node_modules, keeps .wo files, sorted by path relative + to the root). Declarations are collected across every discovered + file before any file's bodies are checked, so symbols span files; + every diagnostic from every stage lands in one Diag.Collector, whose + (file, line, col) sort (diag.ml, Task 2) is what actually gives the + final ordering — not the discovery order files happen to be visited + in. The bare `woc ` form now runs the full pipeline (lex, + parse, typecheck, ownership-check) instead of only checking that the + path exists. 0 = clean compile - 1 = diagnostics reported (reachable now: --dump-tokens on source - with an unknown character, or --dump-ast on source with a - broken declaration, reports WO-E0xx/WO-E1xx and exits 1) + 1 = diagnostics reported 2 = usage or IO failure - With no arguments, this prints usage to stderr and exits 2. Given a - single path argument (no flag), it exits 0 if the path exists and 2 - (with a stderr message) if it does not — this bare-path form - predates any real compilation and stays as-is until Task 8. *) + With no arguments, this prints usage to stderr and exits 2. + + ---- Multi-file dump layout ---- + + For a single discovered file, every --dump-* flag's stdout output is + byte-identical to before Task 8 (no header, nothing changed). When a + path resolves to more than one file, each file's dump is preceded by + a Woc_lib.Dump.file_header line naming that file, and files appear + in the same sorted discovery order used everywhere else — see + compiler/src/dump.ml's doc comment on file_header for the exact + format. *) let usage_msg = "usage: woc \n\ - usage: woc --dump-tokens \n\ - usage: woc --dump-ast \n\ + usage: woc --dump-tokens \n\ + usage: woc --dump-ast \n\ + usage: woc --dump-owner \n\ \n\ Compiles writeonce (.wo) source. is a single .wo file or a\n\ - directory to discover .wo files under (directory discovery lands in\n\ - a later task; compilation itself has not landed yet either).\n\ + directory: a directory is discovered recursively for every .wo file\n\ + under it (dot-prefixed entries and target/data/node_modules are\n\ + skipped, same as `wo run`), sorted by path so discovery order is\n\ + deterministic. Multiple discovered files compile as one program —\n\ + declarations in one file are visible to bodies in another.\n\ + \n\ + With no flag, is fully compiled (lexed, parsed, typechecked,\n\ + ownership-checked) and nothing is printed on success; diagnostics,\n\ + if any, print to stderr.\n\ \n\ --dump-tokens prints one line per lexed token to stdout, in source\n\ order (\"LINE:COL KIND\" or \"LINE:COL KIND(payload)\"), ending with\n\ @@ -41,6 +64,16 @@ let usage_msg = statement/expression parsing inside method and free-fn bodies);\n\ parsing diagnostics, if any, print to stderr.\n\ \n\ + --dump-owner runs the lexer, parser, typechecker and ownership pass,\n\ + then prints the ownership pass's four emitter tables to stdout in\n\ + source order (moves, scope-end drops, @gc rc sites, residual borrow\n\ + sites — see compiler/src/dump.ml for the format); lexing, parsing,\n\ + type and ownership (WO-E3xx) diagnostics print to stderr.\n\ + \n\ + For a directory (or otherwise multi-file) path, every --dump-* flag\n\ + prints each file's own dump in turn, separated by a header line — see\n\ + compiler/src/dump.ml's file_header doc comment.\n\ + \n\ Exit codes: 0 clean, 1 diagnostics reported, 2 usage/IO failure.\n" let read_source path = @@ -52,59 +85,238 @@ let read_source path = Ok s with Sys_error msg -> Error msg -let dump_tokens path = - if not (Sys.file_exists path) then begin - Printf.eprintf "woc: no such file or directory: %s\n" path; - exit 2 - end; - match read_source path with +(* ---- File discovery (Task 8) ---------------------------------------- + + Mirrors wo run's discovery contract (crates/rt/src/lib.rs::discover) + exactly: recursive, skips any entry (file or directory) whose name + starts with '.' or is literally "target", "data", or "node_modules", + keeps only ".wo"-suffixed files, and returns them sorted by path + relative to the root directory (so nested directories don't disturb + the order a flat listing would give). A single file argument is + returned as the one-element list [path], no filtering applied — that + matches the bare-path form's pre-Task-8 behavior of accepting + whatever file it's given. *) + +let skip_name (name : string) : bool = + (String.length name > 0 && name.[0] = '.') + || name = "target" || name = "data" || name = "node_modules" + +let discover_dir (root : string) : string list = + let rec walk (dir : string) (rel : string) : (string * string) list = + Sys.readdir dir |> Array.to_list + |> List.concat_map (fun name -> + if skip_name name then [] + else + let full = Filename.concat dir name in + let rel' = if rel = "" then name else Filename.concat rel name in + if Sys.is_directory full then walk full rel' + else if Filename.check_suffix name ".wo" then [ (rel', full) ] + else []) + in + walk root "" |> List.sort (fun (a, _) (b, _) -> compare (a : string) b) |> List.map snd + +let discover_files (path : string) : (string list, string) result = + if not (Sys.file_exists path) then + Error (Printf.sprintf "no such file or directory: %s" path) + else if Sys.is_directory path then Ok (discover_dir path) + else Ok [ path ] + +let discover_and_read (path : string) : (string * string) list = + match discover_files path with | Error msg -> Printf.eprintf "woc: %s\n" msg; exit 2 - | Ok src -> - let collector = Woc_lib.Diag.Collector.create () in - let toks = Woc_lib.Lexer.tokenize collector ~file:path src in - print_string (Woc_lib.Dump.dump_tokens toks); - if Woc_lib.Diag.Collector.has_error collector then begin - let lookup f = if f = path then Some src else None in - prerr_string (Woc_lib.Diag.Collector.render_all collector lookup); - prerr_newline (); - exit 1 - end - else exit 0 + | Ok files -> + List.map + (fun f -> + match read_source f with + | Ok src -> (f, src) + | Error msg -> + Printf.eprintf "woc: %s\n" msg; + exit 2) + files + +let build_lookup (sources : (string * string) list) : Woc_lib.Diag.source_lookup = + let tbl = Hashtbl.create (List.length sources) in + List.iter (fun (f, src) -> Hashtbl.replace tbl f src) sources; + fun f -> Hashtbl.find_opt tbl f + +let finish (collector : Woc_lib.Diag.Collector.t) (lookup : Woc_lib.Diag.source_lookup) : unit = + if Woc_lib.Diag.Collector.has_error collector then begin + prerr_string (Woc_lib.Diag.Collector.render_all collector lookup); + prerr_newline (); + exit 1 + end + else exit 0 + +(* ---- Cross-file symbol resolution (Task 8) --------------------------- + + Types.collect_declarations / Types.typecheck_program are already + split into a declare-pass and a check-pass (Task 6); that split is + exactly what multi-file needs, so the driver spans files by calling + each pass once per file and merging the declare-pass output before + any file's check-pass runs — types.ml itself needs no change. Each + file's own collect_declarations call still gets that file's own + `~file`, so its own diagnostics (e.g. WO-W201) tag the right file; + only the merged `symbols` value, not any single file's, is what + check-pass calls see, so a class declared in one file resolves for a + field/constructor/etc. in another regardless of discovery order. *) + +let parse_all (collector : Woc_lib.Diag.Collector.t) (sources : (string * string) list) : + (string * Woc_lib.Ast.program) list = + List.map + (fun (f, src) -> + let toks = Woc_lib.Lexer.tokenize collector ~file:f src in + let prog = Woc_lib.Parser.parse collector ~file:f toks in + (f, prog)) + sources + +let merge_symbols (syms_list : Woc_lib.Types.symbols list) : Woc_lib.Types.symbols = + let module SM = Woc_lib.Types.StringMap in + let keep_first _key a _b = Some a in + List.fold_left + (fun (acc : Woc_lib.Types.symbols) (s : Woc_lib.Types.symbols) -> + Woc_lib.Types.{ + classes = SM.union keep_first acc.classes s.classes; + interfaces = SM.union keep_first acc.interfaces s.interfaces; + free_fns = SM.union keep_first acc.free_fns s.free_fns; + typedefs = SM.union keep_first acc.typedefs s.typedefs; + modules = acc.modules @ s.modules; + }) + Woc_lib.Types.{ + classes = SM.empty; interfaces = SM.empty; free_fns = SM.empty; + typedefs = SM.empty; modules = []; + } + syms_list + +(* ---- Cross-file symbol collision (review follow-up, Important 2) ----- + + merge_symbols's first-wins StringMap.union was silent: a same-named + class/interface declared again in a later file doesn't disappear — + it's just dropped from the merged table, so that file's own methods + still get typechecked, but against the *winning* file's field list. + That is a real wrong-shape bug (spurious unknown-field/missing-field + on otherwise-correct code, or a silent pass against the wrong + shape), not merely an untested edge. Reported once per collision, at + merge time, before the losing declaration's own position is gone — + the first-wins merge behavior itself is unchanged; only the silence + is fixed. *) + +let duplicate_symbol_code = Woc_lib.Diag.types_prefix ^ "14" (* WO-E214 *) + +let report_collision (collector : Woc_lib.Diag.Collector.t) ~(kind : string) ~(name : string) + ~(file : string) ~(pos : Woc_lib.Ast.pos) ~(first_file : string) + ~(first_pos : Woc_lib.Ast.pos) : unit = + Woc_lib.Ast.( + Woc_lib.Diag.Collector.add collector + (Woc_lib.Diag.error ~code:duplicate_symbol_code ~file ~line:pos.line ~col:pos.col + ~message:(Printf.sprintf "%s `%s` already declared in `%s`" kind name first_file) + ~related: + [ Woc_lib.Diag.related_site ~file:first_file ~line:first_pos.line ~col:first_pos.col + ~label:(Printf.sprintf "`%s` first declared here" name) + ] + ())) + +(* Walks (file, symbols) pairs in discovery order, per kind (class, + interface), remembering the first file/pos to declare each name and + reporting every later redeclaration against it. free_fns/typedefs + aren't checked: nothing downstream resolves them by cross-file + lookup the way class/interface satisfaction does, so a same-name + free fn isn't the silent-wrong-shape hazard this exists for — out of + scope for this fix, not something the review asked for. *) +let check_symbol_collisions (collector : Woc_lib.Diag.Collector.t) + (per_file : (string * Woc_lib.Types.symbols) list) : unit = + let seen_classes : (string, string * Woc_lib.Ast.pos) Hashtbl.t = Hashtbl.create 16 in + let seen_interfaces : (string, string * Woc_lib.Ast.pos) Hashtbl.t = Hashtbl.create 16 in + List.iter + (fun (file, (syms : Woc_lib.Types.symbols)) -> + Woc_lib.Types.( + StringMap.iter + (fun name (c : class_info) -> + match Hashtbl.find_opt seen_classes name with + | Some (first_file, first_pos) -> + report_collision collector ~kind:"class" ~name ~file ~pos:c.pos ~first_file ~first_pos + | None -> Hashtbl.add seen_classes name (file, c.pos)) + syms.classes; + StringMap.iter + (fun name (i : interface_info) -> + match Hashtbl.find_opt seen_interfaces name with + | Some (first_file, first_pos) -> + report_collision collector ~kind:"interface" ~name ~file ~pos:i.pos ~first_file + ~first_pos + | None -> Hashtbl.add seen_interfaces name (file, i.pos)) + syms.interfaces)) + per_file + +let typecheck_all (collector : Woc_lib.Diag.Collector.t) + (parsed : (string * Woc_lib.Ast.program) list) : Woc_lib.Types.symbols = + let per_file_syms = + List.map + (fun (f, prog) -> (f, Woc_lib.Types.collect_declarations ~file:f prog collector)) + parsed + in + check_symbol_collisions collector per_file_syms; + let syms = merge_symbols (List.map snd per_file_syms) in + List.iter + (fun (f, prog) -> Woc_lib.Types.typecheck_program ~file:f prog syms collector) + parsed; + syms + +let dump_tokens path = + let sources = discover_and_read path in + let collector = Woc_lib.Diag.Collector.create () in + let multi = List.length sources > 1 in + List.iter + (fun (f, src) -> + let toks = Woc_lib.Lexer.tokenize collector ~file:f src in + if multi then print_string (Woc_lib.Dump.file_header f); + print_string (Woc_lib.Dump.dump_tokens toks)) + sources; + finish collector (build_lookup sources) let dump_ast path = - if not (Sys.file_exists path) then begin - Printf.eprintf "woc: no such file or directory: %s\n" path; - exit 2 - end; - match read_source path with - | Error msg -> - Printf.eprintf "woc: %s\n" msg; - exit 2 - | Ok src -> - let collector = Woc_lib.Diag.Collector.create () in - let toks = Woc_lib.Lexer.tokenize collector ~file:path src in - let prog = Woc_lib.Parser.parse collector ~file:path toks in - print_string (Woc_lib.Dump.dump_ast prog); - if Woc_lib.Diag.Collector.has_error collector then begin - let lookup f = if f = path then Some src else None in - prerr_string (Woc_lib.Diag.Collector.render_all collector lookup); - prerr_newline (); - exit 1 - end - else exit 0 + let sources = discover_and_read path in + let collector = Woc_lib.Diag.Collector.create () in + let multi = List.length sources > 1 in + let parsed = parse_all collector sources in + List.iter + (fun (f, prog) -> + if multi then print_string (Woc_lib.Dump.file_header f); + print_string (Woc_lib.Dump.dump_ast prog)) + parsed; + finish collector (build_lookup sources) + +let dump_owner path = + let sources = discover_and_read path in + let collector = Woc_lib.Diag.Collector.create () in + let multi = List.length sources > 1 in + let parsed = parse_all collector sources in + let syms = typecheck_all collector parsed in + List.iter + (fun (f, prog) -> + let tables = Woc_lib.Owner.analyze ~file:f prog syms collector in + if multi then print_string (Woc_lib.Dump.file_header f); + print_string (Woc_lib.Dump.dump_owner tables)) + parsed; + finish collector (build_lookup sources) + +(* The bare `woc ` form (Task 8): runs the full pipeline with no + dump — check-only. Nothing is printed to stdout on success, matching + the exit-code contract's "0 = clean compile". *) +let check_only path = + let sources = discover_and_read path in + let collector = Woc_lib.Diag.Collector.create () in + let parsed = parse_all collector sources in + let syms = typecheck_all collector parsed in + List.iter (fun (f, prog) -> ignore (Woc_lib.Owner.analyze ~file:f prog syms collector)) parsed; + finish collector (build_lookup sources) let () = match Sys.argv with | [| _; "--dump-tokens"; path |] -> dump_tokens path | [| _; "--dump-ast"; path |] -> dump_ast path - | [| _; path |] -> - if Sys.file_exists path then exit 0 - else begin - Printf.eprintf "woc: no such file or directory: %s\n" path; - exit 2 - end + | [| _; "--dump-owner"; path |] -> dump_owner path + | [| _; path |] -> check_only path | _ -> prerr_string usage_msg; exit 2 diff --git a/compiler/src/diag.ml b/compiler/src/diag.ml index cc2efc2..580c5d8 100644 --- a/compiler/src/diag.ml +++ b/compiler/src/diag.ml @@ -58,6 +58,7 @@ let lexing_prefix = "WO-E0" let parsing_prefix = "WO-E1" let types_prefix = "WO-E2" let ownership_prefix = "WO-E3" +let warning_prefix = "WO-W" type severity = | Error diff --git a/compiler/src/dump.ml b/compiler/src/dump.ml index 632ccc1..b5a22ba 100644 --- a/compiler/src/dump.ml +++ b/compiler/src/dump.ml @@ -77,6 +77,16 @@ let kind_label (k : Token.kind) : string = | Token.Newline -> "NEWLINE" | Token.Eof -> "EOF" +(* Multi-file dump layout (Task 8, bin/main.ml). Every dump_* function + above still renders exactly one file's contribution — that contract + doesn't change. When a --dump-* flag's resolves to more than + one discovered file, main.ml prints each file's dump_* output one + after another in sorted discovery order, each preceded by this + separator line naming the file. For a single file, main.ml never + calls this, so every dump's stdout stays byte-identical to every + pre-Task-8 golden fixture. *) +let file_header (path : string) : string = Printf.sprintf "=== %s ===\n" path + let dump_tokens (toks : Token.t list) : string = let lines = List.map @@ -273,3 +283,152 @@ let dump_decl : Ast.decl -> string list = function let dump_ast (prog : Ast.program) : string = let lines = List.concat_map dump_decl prog.decls in match lines with [] -> "" | _ -> String.concat "\n" lines ^ "\n" + +(* dump_owner — the ownership pass's four emitter tables (Task 7). + + Four fixed sections in a fixed order, each listing its entries in + source order (LINE:COL first, same convention as every other dump + here); a section with no entries still prints its header, so the + format is stable and a golden diff shows an emptied table as a real + change. Node ids are not printed — the tables carry them for plan 3 + (Owner.move_site.mv_node and friends), but ids churn goldens exactly + the way dump_ast's module doc describes, so positions are the + rendering surface. + + == MOVES == one line per real ownership transfer + "LINE:COL MOVE ", where is + LET / ASSIGN / RETURN / ARG(param) / CTOR(field). + Copy-classed and @gc-classed transfers are absent + by design (a register copy and an rc site + respectively, not a transfer). + + == DROPS == deterministic destruction, plus the frame's drop + map at trap-capable sites: + SCOPE