shoney.arickathil
335a797479
Phased plan argued from the 2026-08-11 spec: Phase 1 additive inference + --dump-gc (golden-neutral), Phase 2 demand promotion + rewire field kinds to the inferred set (+ @gc-in-source error), Phase 3 the runtime collector swap (header rewrite, traced list, tri-color mark + Yuasa barrier, retire RC, .wob bump), Phase 4 doc migration. Each phase has file targets + verify commands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
152 lines
12 KiB
Markdown
152 lines
12 KiB
Markdown
# Inferred GC + incremental mark-sweep — implementation plan
|
||
|
||
> **For agentic workers:** use superpowers:executing-plans (inline) or
|
||
> subagent-driven-development. Steps are checkboxes. Per repo rule, this plan
|
||
> carries **actions in words + verification commands, no code blocks** — the
|
||
> design detail lives in the spec, which travels with this plan.
|
||
|
||
**Goal:** Replace developer-annotated `@gc` + reference counting with
|
||
compiler-**inferred** GC-ness and an incremental per-shard tri-color
|
||
**mark-sweep** collector, so the developer writes no memory annotations and the
|
||
whole `@gc`-RC bug class is deleted by construction.
|
||
|
||
**Architecture:** A new compiler pass classifies each class `owned` or `gc`
|
||
(structural SCC over the class-reference graph + demand promotion from the
|
||
ownership pass), feeding the single existing seam `Types.is_gc_class`. The
|
||
runtime retires RC + the Bacon–Rajan trial-deletion collector and grows a
|
||
per-shard traced object list swept by a tri-color marker with a Yuasa deletion
|
||
barrier; the 16-byte object header is repurposed (rc+borrow → sweep-list link),
|
||
so nothing grows.
|
||
|
||
**Tech stack:** OCaml stdlib only (compiler, no opam); C11 + libc only
|
||
(runtime); dune + make + `just`; golden + corpus + ASan gates.
|
||
|
||
**Spec:** [`../specs/2026-08-11-inferred-gc-mark-sweep-design.md`](../specs/2026-08-11-inferred-gc-mark-sweep-design.md)
|
||
(read it — every task argues from it). Worked example + diagrams:
|
||
[`../../examples/gc-cycle/README.md`](../../examples/gc-cycle/README.md).
|
||
|
||
## Global Constraints
|
||
|
||
- OCaml stdlib only; no Menhir/ppx/opam. C11, libc only, direct syscalls.
|
||
- The object header stays **exactly 16 bytes** (`_Static_assert(sizeof(wo_hdr)==16)` must survive).
|
||
- `ref B` is an id (`Copy`) and creates **no** class-graph edge — never traced by structure.
|
||
- Land inference **and** the collector together as one coherent change set (dropping `@gc` without replacing RC re-opens the exact bug class it caused). Phases below may commit incrementally, but the branch is not "done" until the collector is in and RC is gone.
|
||
- Every GC-ness promotion is **reported** (a note); `@gc` in source becomes a **diagnostic**, never silently accepted.
|
||
- Collection is **per-shard**; no global stop-the-world, no cross-shard tracing, non-moving (no compaction).
|
||
- Verification gates, unchanged in name: `just woc-test`, `just wovm-test`, `just oop-e2e`, `just oop-accept`; goldens re-blessed with `WOC_BLESS=1 dune runtest --root compiler`.
|
||
|
||
---
|
||
|
||
## Phase 1 — Inference pass, additive (no emit/runtime change)
|
||
|
||
Deliverable: `woc --dump-gc <path>` prints the classification; **no emitted
|
||
bytecode, golden, or runtime behavior changes yet** (field kinds still derive
|
||
from the annotation, so existing goldens are untouched). Lowest-risk landing.
|
||
|
||
### Task 1.1 — class-reference graph + Tarjan SCC (`compiler/src/gcinfer.ml`)
|
||
|
||
**Files:** Create `compiler/src/gcinfer.ml`; Modify `compiler/src/dune` (add `gcinfer` to `modules`).
|
||
|
||
- [ ] Build a directed graph over class names: edge `A → B` when `A` has a field whose resolved type is `B`, `?B`, `multi B`, `map<B,_>`, or `map<_,B>` (unwrap `Nullable`). **`ref B` contributes no edge.** Read fields from `Ast` class decls; resolve names via the symbol table `types.ml` already builds.
|
||
- [ ] Run Tarjan's SCC (hand-written, stdlib only). Classify a class **traced** iff it is in a non-trivial SCC **or** has a self-loop; else **owned**. Expose `Gcinfer.classify : <syms/classes> -> result` returning the traced-name set plus, per traced class, the reason (a cycle path for the note).
|
||
- [ ] Verify with a tiny OCaml unit in `compiler/test` (or the existing runner) over: self-loop (`Node.next: ?Node`), mutual recursion (`A.b:B`, `B.a:A`), `multi Self`, `map<_,Self>`, and the `ref T`-creates-no-edge case. Run `just woc-test`; expected PASS.
|
||
- [ ] Commit.
|
||
|
||
### Task 1.2 — `--dump-gc` mode + golden
|
||
|
||
**Files:** Modify `compiler/bin/main.ml` (argv dispatch + usage), `compiler/src/dump.ml` (renderer). Test: new golden under `compiler/test/golden/`.
|
||
|
||
- [ ] Add a `--dump-gc <path>` mode: run lex→parse→types, call `Gcinfer.classify`, print one `Name<pad>owned|gc<pad>(reason)` line per class in declaration order (the spec's `--dump-gc` artifact shape). Reason is the cycle path for structural, empty for owned.
|
||
- [ ] Add the usage line and the mode to the dispatch match (beside `--dump-owner`).
|
||
- [ ] Add a golden fixture: run `--dump-gc` over `docs/examples/gc-cycle` (expect `Node gc (cycle Node -> Node)`, `Segment owned`) and over the pricing corpus subset. Bless with `WOC_BLESS=1`.
|
||
- [ ] Run `just woc-test`; expected PASS, and **existing goldens unchanged** (no emit path touched). Commit.
|
||
|
||
---
|
||
|
||
## Phase 2 — Demand promotion + rewire field kinds to the inferred set (still on RC runtime)
|
||
|
||
Deliverable: field GCREF-ness and owner exemptions derive from inference, not
|
||
the annotation; `@gc` in source is an error; the gc-cycle sample **compiles and
|
||
runs on today's Bacon–Rajan collector**. Goldens re-blessed. This proves the
|
||
front-end end to end before the collector swap.
|
||
|
||
### Task 2.1 — demand half (promotion from ownership escapes)
|
||
|
||
**Files:** Modify `compiler/src/gcinfer.ml`, `compiler/src/owner.ml` (add a collect-promotions mode).
|
||
|
||
- [ ] Add a mode to the ownership pass that, instead of emitting `WO-E304`/long-lived-alias errors, records the offending class. `Gcinfer` runs owner in this mode, unions the recorded classes into the traced set, and re-runs — terminating because the set only grows (bounded by class count). Each demand promotion carries its escape-site note.
|
||
- [ ] Verify: a `PriceCache`-shaped fixture (acyclic, aliased) classifies `gc (alias escape, …)` via `--dump-gc`. `just woc-test` PASS. Commit.
|
||
|
||
### Task 2.2 — inference is the source of GC-ness; annotation errors
|
||
|
||
**Files:** Modify `compiler/src/types.ml` (`is_gc_class` reads the inferred set), `compiler/src/parser.ml` (`@gc` arm → diagnostic), `compiler/src/dump.ml` (stop rendering ` @gc`), `compiler/src/emit.ml`/`disasm.ml` (class-flag provenance only; bit unchanged). Error: new WO-E1xx in `docs/plan/oop-vm/01-error-catalog.md`.
|
||
|
||
- [ ] Thread the inferred traced-set into the typing context so `Types.is_gc_class` answers from it. Field-kind derivation (→ `WO_K_GCREF`) and every `owner.ml` exemption then follow with no further change (that is the seam).
|
||
- [ ] Turn the parser's `@gc` acceptance into a WO-E1xx diagnostic pointing at inference + `--dump-gc`. Update the error catalog.
|
||
- [ ] Re-bless every golden that rendered ` @gc`, `flags=gc`, or a changed GCREF field kind (`WOC_BLESS=1`). Convert the `gc/` corpus fixtures to drop `@gc` from source (they rely on inference now).
|
||
- [ ] Verify: `docs/examples/gc-cycle` now **emits** (no WO-E301 — traced classes alias freely) and **runs** on the current runtime, printing `ring a -> b -> c -> a`. `just woc-test`, `just oop-e2e` PASS. Commit.
|
||
|
||
---
|
||
|
||
## Phase 3 — Runtime collector swap (the coherent landing)
|
||
|
||
Deliverable: RC and trial deletion are gone; the incremental tri-color
|
||
mark-sweep with the Yuasa barrier reclaims cycles; `.wob` version bumped. This
|
||
is the largest phase and lands with Phase 2's front-end.
|
||
|
||
### Task 3.1 — header rewrite + per-shard traced list
|
||
|
||
**Files:** Modify `runtime/src/wob.h` (`wo_hdr`), `runtime/src/obj.h` (`wo_rt` list head + `wo_obj_new` links traced objects), `runtime/src/gc.{c,h}`.
|
||
|
||
- [ ] Repurpose the header: retire `rc` and (for traced objects) `borrow`; give those 8 contiguous bytes to a 64-bit intrusive sweep-list link. Keep colors in the existing `WO_F_COLOR` bits. The `_Static_assert(sizeof(wo_hdr)==16)` must still hold.
|
||
- [ ] `wo_rt` gains a traced-list head; `wo_obj_new` links a traced-class instance (class-flag bit set) in as white. Sweep recovers size via `wo_obj_size` (class table). Retire `cycbuf` and `WO_F_BUF`.
|
||
- [ ] Verify build both dispatch flavors: `just wovm-build` + `make -C runtime test test-iso`. Commit.
|
||
|
||
### Task 3.2 — tri-color incremental mark + Yuasa barrier + budgeted sweep
|
||
|
||
**Files:** Modify `runtime/src/gc.{c,h}`, `runtime/src/vm.c` (roots via pc gc-mask; barrier in `SETF`/`map_set`/`push`; retire `RC_INC`/`RC_DEC` cases; safepoints at back-edges/calls).
|
||
|
||
- [ ] Delete trial deletion (`mark_gray`/`scan_black`/`scan_`/`collect_white`/`white_free`/zombie guard). Implement: roots = value/frame slots read via the per-pc gc-mask; grey worklist; mark budget `WO_GC_BUDGET`; owned objects traversed-not-freed, skipping subtrees via the precomputed "transitively-contains-gcref" class bit; sweep frees white + unlinks, repaints black→white; heap-goal trigger; `WO_GC_TRACE` per-slice counts.
|
||
- [ ] Yuasa deletion barrier: on a store into a `GCREF` slot **while marking**, shade the old value grey. Lives in the VM store paths (no new opcode).
|
||
- [ ] Verify: `just wovm-test`. Commit.
|
||
|
||
### Task 3.3 — emitter + format: retire RC, restate the gc-mask, bump `.wob`
|
||
|
||
**Files:** Modify `compiler/src/emit.ml` (drop `emit_rc` + escape-acquire anchor; gc-mask now = GC roots), `compiler/src/owner.ml` (delete rc table/elision/`resolve_rc`/`release_gc`/clobber rule), interpreter (opcodes 27–28 reserved), `docs/plan/oop-vm/00-wob-format.md` (version bump + reserved opcodes + gc-mask contract), `runtime/src/loader.c` if it validates opcodes.
|
||
|
||
- [ ] Stop emitting `RC_INC`/`RC_DEC`; reserve the opcodes; bump the `.wob` version in the format doc + loader constant. Restate the drop-table gc-mask contract as "GC roots at this pc". `wo_drop_kind` for `WO_K_GCREF` becomes a no-op.
|
||
- [ ] Re-bless all affected goldens (`--dump-bc`, `--dump-owner`, disasm) with `WOC_BLESS=1`.
|
||
- [ ] Verify: `just woc-test`, `just oop-e2e`, `just oop-accept`. Commit.
|
||
|
||
### Task 3.4 — fixtures: adversarial barrier + cycle rewrites
|
||
|
||
**Files:** Modify `runtime/test/test_cycle.c`, `runtime/test/test_rc.c`; corpus `tests/corpus/gc/{abandoned-cycle,budget-steps,held-cycle}`; add an adversarial barrier fixture.
|
||
|
||
- [ ] Rewrite `test_cycle.c`/`test_rc.c` off rc assertions onto: abandoned cycle freed, rooted cycle survives, slices bounded, sweep-list leak-free after N cycles. Re-bless `abandoned-cycle`/`budget-steps` traces; **redefine** `held-cycle` as "a cycle rooted from a live frame survives a slice" (from inside a running program).
|
||
- [ ] Add the barrier fixture: hide a traced object between slices (store into a blackened object, drop the original ref); it must survive with the barrier in and be freed (corruption) with it compiled out — the design's safety net.
|
||
- [ ] Verify: `just wovm-test`, `just oop-accept`, ASan clean after repeated cycles. Commit.
|
||
|
||
---
|
||
|
||
## Phase 4 — Migration: amend the normative docs
|
||
|
||
**Files:** `docs/00-principles.md` (principle 3), the OOP spec (decision table, §3 rule 5, §4 memory model), `docs/plan/oop-vm/00-wob-format.md`, `01-error-catalog.md` (retire WO-W201, update WO-E304 wording, add the new WO-E1xx), `08-builtin-surface.md` (delete the `push` special case + `set` gap), `docs/00-status.md` (record 7b superseding iteration 2's memory model), `docs/stories/language-runtime-database/07b-inferred-gc-mark-sweep.md` (status → done), `docs/examples/gc-cycle/README.md` (flip "Run status" to shipped + wire a `just gc-cycle` acceptance).
|
||
|
||
- [ ] Apply each amendment in the spec's §8 migration table. Add a `docs/examples/gc-cycle` acceptance script + `just gc-cycle` recipe running `--dump-gc` + ring/owned demos under `WO_GC_TRACE`.
|
||
- [ ] Verify: `just oop-accept` green; `just gc-cycle` green; `git grep '@gc' -- '*.wo'` returns nothing (success criterion 1). Commit.
|
||
|
||
---
|
||
|
||
## Success criteria (from the spec §Success criteria)
|
||
|
||
1. No `.wo` contains `@gc`; using it is a diagnostic.
|
||
2. `--dump-gc` classifies every class; each traced reason is a cycle path or a named escape.
|
||
3. `RC_INC`/`RC_DEC` in no emitted image; opcodes reserved in the format doc.
|
||
4. The adversarial barrier fixture fails barrier-out, passes barrier-in.
|
||
5. An abandoned cycle collects within budgeted slices; a rooted cycle survives.
|
||
6. Whole corpus ASan-clean, including after repeated cycles — closing milestone-1 criterion 3.
|
||
|
||
## Self-review notes
|
||
|
||
- Spec coverage: §1 inference → Ph1+2.1; §2 emitter → 2.2/3.3; §3 header+list → 3.1; §4 mark+barrier → 3.2; §5 error handling → inherent (no new mechanism); §6 deletions → 3.2/3.3; §7 testing → 3.4; §8 migration → Ph4. No gaps.
|
||
- Order risk: Phase 1 is provably golden-neutral (no emit path touched). Phase 2 first changes emitted output (field kinds) → re-bless. Phase 3 is the format bump → biggest re-bless. Phases 2+3 must ship together to satisfy the "coherent landing" constraint even though committed as steps.
|