From 0b87c6c5f35637e7ab54e683fcf6f58d785b1bca Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Sat, 22 Aug 2026 21:18:33 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20story=2036=20+=20plan=20=E2=80=94=20ope?= =?UTF-8?q?rator=20parity=20(not,=20bitwise,=20hex=20literals,=20compound?= =?UTF-8?q?=20assigns)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - story: gap survey vs Go reference; forks settled (arithmetic >>, trap on out-of-range shift count, Lua-placement not) - plan: 5 tasks, lexer -> parser -> checker -> emit/VM (.wob v6) -> closeout - latent defect recorded: += / -= lex but never parse (dead tokens) Co-Authored-By: Claude Fable 5 --- .../refine/36-operator-parity.md | 155 +++++++++++ .../plans/2026-08-22-operator-parity.md | 261 ++++++++++++++++++ 2 files changed, 416 insertions(+) create mode 100644 docs/stories/language-runtime-database/refine/36-operator-parity.md create mode 100644 docs/superpowers/plans/2026-08-22-operator-parity.md diff --git a/docs/stories/language-runtime-database/refine/36-operator-parity.md b/docs/stories/language-runtime-database/refine/36-operator-parity.md new file mode 100644 index 0000000..ad2962e --- /dev/null +++ b/docs/stories/language-runtime-database/refine/36-operator-parity.md @@ -0,0 +1,155 @@ +--- +iteration: "36" +status: refine +--- + +# Iteration 36 — operator parity: `not`, bitwise, hex literals, compound assigns + +> Format: fiberloom `product/story-iteration-template`. Part of +> [Story — one language, one runtime, one database, one binary](../00-story.md). +> +> **Inserted 2026-08-22, forks settled the same day** (decisions below, +> story-19 discipline). Driver: a gap survey of the operator surface +> against the Go reference checkout (`.dev/reference/go` — +> `src/go/token/token.go`, spec §Arithmetic operators) found four holes +> that every mainstream language covers. The survey also found a latent +> defect: `+=`/`-=` are lexed (`lexer.ml:541/548`, tokens at +> `token.ml:149-150`) but no parser rule consumes them — dead tokens +> since the haxe-parity control-surface task, zero uses in the corpus, +> so `x += 1` today dies as a generic WO-E101 instead of working or +> being honestly absent. +> +> The driving consumer is iteration 34 (crypto builtins): HMAC's +> ipad/opad step is a byte-wise XOR, and the doctrine question recorded +> there — C-side builtin vs pure-`.wo` — is unanswerable while the +> language cannot spell XOR at all. Secondary consumer: `http/auth.wo`'s +> pure-`.wo` base64 does with division/modulo what shifts and masks say +> directly. +> +> **Plan:** [`2026-08-22-operator-parity.md`](../../../superpowers/plans/2026-08-22-operator-parity.md) +> (5 tasks: lexer/tokens; AST/parser incl. compound-assign desugar; +> type checker incl. literal shift-count rejection; emit + VM opcodes +> `.wob` v6; corpus/gates/docs closeout). + +## Why this iteration exists + +The expression grammar (`parser.ml` ~594, `token.ml:134-150`) stops at: +arithmetic `+ - * / %`, concat `..`, comparison `== != < <= > >=`, +logical `and`/`or`, unary minus. Four common gaps, in priority order: + +1. **No boolean negation.** No `not`, no `!`, no unary NOT node in the + AST. The only spelling is `x == false` — every caller pays, every + day. Highest value for smallest surface. +2. **No bitwise operators.** No `&`, `|`, `^`, `<<`, `>>` in any form. + Blocks iteration 34's HMAC honestly; base64/multipart boundary code + emulates masks with arithmetic. +3. **No hex/binary integer literals.** The lexer scans decimal digit + runs only (`lexer.ml:386` on). Bitwise code without `0xFF` masks is + write-only. +4. **Compound assigns half-promised.** `+=`/`-=` lex but never parse + (the latent defect above); `*=` `/=` `%=` do not exist at all. + +Deliberately NOT gaps (doctrine, not omission): `&&`/`||`/`!` symbol +forms (words won — KwAnd/KwOr precedent), ternary `?:` (switch is +already an expression, `parser.ml:1181`), `++`/`--`, function values, +inheritance-family keywords (WO-E105 rejects them by name). + +## Goals + +- **`not` keyword**, unary boolean, sitting with unary minus in the + grammar (settled below: Lua placement, binds tighter than comparison). + Bool-only operand, diagnosed like every other type error. +- **Five bitwise operators on Int only**: `&` AND, `|` OR, `^` XOR, + `<<` left shift, `>>` right shift. Precedence copies Go's C-trap fix + (`token.go:266-280`): `&`/`<<`/`>>` at the multiplicative level, + `|`/`^` at the additive level — both ABOVE comparison, so + `x & mask == 0` groups the AND first. Complement is spelled + `-1 ^ x` (single signed 64-bit Int makes Go's mask rule collapse to + exactly this); no `~`, no unary `^`, no `&^`. +- **Hex and binary literals**: `0x` and `0b` prefixes on Int, plus `_` + digit separators in all integer forms. Decimal lexing stays + byte-identical for every existing fixture (the iteration-19 + discipline). +- **Compound assigns wired**: `+=` `-=` parse into the existing + assignment statement (resurrecting the dead tokens), `*=` `/=` `%=` + join them. Statement-level sugar over the existing place logic — + no new AST evaluation semantics. +- **Runtime**: new i64 register opcodes beside `WOP_ADD..WOP_NEG` + (`runtime/src/wob.h:194`) for AND/OR/XOR/SHL/SHR (+NOT if the emitter + wants it); `.wob` version bump, loader validation, disasm coverage — + the v5 (Float/Bytes) change is the template. + +## Settled decisions (2026-08-22) + +1. **`>>` is arithmetic (sign-extending).** Go's own choice for signed + integers, and writeonce has exactly one Int, signed 64-bit. For the + driving consumers the fork is moot anyway — crypto/base64 shift + byte-range values whose sign bit is never set, where arithmetic and + logical are bit-identical. No logical-shift operator ships; a caller + who wants zero-fill masks first. +2. **Out-of-range shift count traps, like DIV0.** Valid counts are + 0..63; a negative or ≥64 count at run time is a named trap on the + same machinery as `WOP_DIV`'s DIV0 (`wob.h:197`) — the established + Int honesty precedent (story 19 kept the trap deliberately). NOT + Go's saturate-to-0/-1 (spec surface serving generic-width code + writeonce doesn't have) and NOT hardware masking (x86's count%64 + makes `x << 64 == x`, the classic silent wat). Real code shifts by + literals: when the count is a literal, the compiler rejects it at + emit time and the trap never runs. +3. **`not` sits at the unary level, beside minus (Lua placement).** + `not a == b` parses `(not a) == b`. The grammar's ordering is + already anchored to Lua by name (`parser.ml` ~601 "This ordering + matches Lua's"), and joining `parse_unary` adds zero new precedence + rungs. Python's looser placement reads closer to English, but it + buys a new rung to prevent a misparse that Bool-only typing already + converts into a compile error in every mixed-type case; the one + silent case (Bool compared to Bool through `not`) gets a pinned + fixture so the choice stays visible. `if not done` and `while not + empty()` — the actual daily uses — read identically under both. + +## Acceptance Criteria + +- **Given** the corpus, **when** fixtures land for `not` (plain, + chained with and/or, non-Bool operand diagnosed), each bitwise + operator, precedence pins (`x & mask == 0` and one shift-vs-additive + case), hex/binary/underscore literals (value-identical to decimal + twins), out-of-range shift (literal count rejected at compile time, + variable count trapping at run time), the `not a == b` grouping pin, + and all five compound assigns on let/field/index places, **then** + corpus passes with zero regressions on every pre-existing fixture. +- **Given** iteration 34's ipad/opad step written in pure `.wo` with + `^` over `byte_at` values, **when** it runs against an RFC 2104 test + vector's intermediate, **then** the bytes match — the consumer that + motivated the iteration is demonstrably unblocked. +- **Given** the existing gates (`just corpus`-equivalent, `just + web-app`, `just deps-accept`, `test_wal`), **when** the iteration + lands, **then** all pass unchanged — the `.wob` bump breaks no + replay/loader path. +- **Given** `x += 1` in a fixture today-vs-after, **when** compiled, + **then** the before is the recorded WO-E101 and the after executes — + the dead-token defect is provably closed, not papered over. + +## Out Of Scope + +Unary complement operator (spell `-1 ^ x`), Go's `&^` AND-NOT, bitwise +compound assigns (`&=` family), `++`/`--`, ternary, symbol forms +`&&`/`||`/`!`, bitwise on Float/Bytes/Bool, overflow-checked arithmetic +changes, octal literals. + +## Proposed Solution + +Lexer: extend the operator switch (the `+`/`+=` two-char pattern at +`lexer.ml:541-548` is the template) for the five operators and the +three new compound assigns; extend the digit scanner with `0x`/`0b` +prefix branches and `_` skipping, leaving the bare-decimal path +untouched. Parser: two new precedence rungs threaded into the existing +recursive-descent chain exactly where Go's table says; `not` joins +`parse_unary`; compound assigns join the statement that already +matches `Token.Eq` (`parser.ml:510`). Types: all five operators and +`not` are `Int -> Int -> Int` / `Bool -> Bool` in the same checker +table that owns `+`. Emit/runtime: new opcodes in `wob.h`'s i64 block, +`WOB_VERSION` bump, interpreter cases beside the existing wrapping +arithmetic, disasm names. The forks are settled above, so no separate +spec is owed (the story-19 deviation precedent: format decisions +recorded normatively where they land, reasoning in CODE-LOGIC.md); the +plan follows once this story is approved on the board. diff --git a/docs/superpowers/plans/2026-08-22-operator-parity.md b/docs/superpowers/plans/2026-08-22-operator-parity.md new file mode 100644 index 0000000..cbf2332 --- /dev/null +++ b/docs/superpowers/plans/2026-08-22-operator-parity.md @@ -0,0 +1,261 @@ +# Iteration 36 — operator parity (`not`, bitwise, hex literals, compound assigns): implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> superpowers:subagent-driven-development (recommended) or +> superpowers:executing-plans to implement this plan task-by-task. +> Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** the language gains boolean `not`, the five Int bitwise +operators (`&` `|` `^` `<<` `>>`), hex/binary/underscore integer +literals, and working compound assigns (`+=` `-=` `*=` `/=` `%=`) — +full stack, `.wob` v6. + +**Architecture:** front-to-back in dependency order — tokens/lexer, +then AST/parser, then the type checker, then emit + VM opcodes, then +the corpus/gates closeout. Every semantic decision is already settled +in the story ("Settled decisions", 2026-08-22): `>>` arithmetic, +out-of-range shift count traps (literal counts rejected at compile +time), `not` at the unary level beside minus. + +**Tech stack:** OCaml compiler (`compiler/src/`), C runtime/VM +(`runtime/src/`), dune + just gates. + +**Spec:** the story IS the spec — +`docs/stories/language-runtime-database/refine/36-operator-parity.md` +(story-19 deviation precedent: decisions recorded normatively there, +reasoning-under-the-code lands in CODE-LOGIC.md as part of this plan). + +## Global constraints + +- Plans in this repo carry no code blocks — each step names the exact + file/anchor and describes the change; the executor writes the code. +- Every pre-existing fixture must lex, parse, and run byte-identically + (the iteration-19 discipline). Any golden that changes is a defect in + the change, not a bless-and-move-on. +- Precedence copies Go (`.dev/reference/go/src/go/token/token.go:266`): + `&` `<<` `>>` join the multiplicative rung, `|` `^` join the additive + rung — no new rungs, both above comparison. +- Bitwise is Int-only. No Float, Bool, Text, Bytes operands — the + existing operand-type diagnostics reject them; `numeric_world` + (`types.ml:193`) is not touched. +- Run gates via just recipes only (`just woc-test`, `just wovm-test`, + `just oop-accept`, `just web-app`, `just deps-accept`). +- Branch off master first (never commit on master); bullet-style commit + messages, ≤25 lines, no push. + +--- + +## Task 1 — lexer and tokens: the new surface exists as tokens + +**Files:** modify `compiler/src/token.ml` (kind list ~134-150), +`compiler/src/lexer.ml` (keyword table ~131-147, operator switch +~500-570, digit scanner ~386-430), `compiler/src/dump.ml` (kind labels +~103). + +**Interfaces produced:** tokens `Amp`, `Caret`, `Shl`, `Shr`, +`StarEq`, `SlashEq`, `PercentEq`, keyword `KwNot`; hex (`0x`), binary +(`0b`), and underscore-separated integer literals arriving as ordinary +`Token.Int`. + +- [ ] Add the seven operator tokens and `KwNot` to `token.ml`'s kind + list, each with the one-line comment style the file already uses; + add `not` to the lexer keyword table beside `and`/`or` + (`lexer.ml:144-145`) — grep the corpus and samples first to confirm + `not` is used nowhere as an identifier (the discipline every keyword + above it followed). +- [ ] Extend the operator switch: new `&` and `^` cases (single-char, + today they fall to the error branch); `*`, `/`, `%` gain a peek for + `=` (the `+`/`+=` two-char pattern at `lexer.ml:543-549` is the + template); `<` gains a peek for a second `<` (order: `=` first so + `<=` stays `LtEq`, then `<`), `>` symmetrically. No `<<=`/`>>=` — + bitwise compound assigns are out of scope per the story. +- [ ] Extend the digit scanner: after an initial `0`, a peek for `x` + or `b` switches to hex/binary accumulation (case-insensitive hex + digits; at least one digit required, else the existing malformed- + number path); `_` is skipped between digits in every integer form + (leading/trailing/doubled `_` falls out as an error naturally). + Accumulation wraps in i64 like the VM's own arithmetic (the ADD + doctrine, `wob.h:194`). The bare-decimal and float paths stay + byte-identical — the iteration-19 comment at `lexer.ml:387` explains + exactly which branch keeps them safe. +- [ ] Add dump labels for the eight new kinds beside PLUSEQ/MINUSEQ + (`dump.ml:103`). +- [ ] Verify: `just woc-test` — every existing golden unchanged (new + tokens are unreachable from old source). Commit. + +## Task 2 — AST and parser: the grammar accepts what the tokens spell + +**Files:** modify `compiler/src/ast.ml` (unop at :149, binop at :158), +`compiler/src/parser.ml` (ladder doc ~594, additive/multiplicative +~860-905, unary at :908, assign statement at :1489, an id-freshening +clone helper beside `subst_expr` ~2068). + +**Interfaces produced:** `unop` gains `Not`; `binop` gains `BAnd`, +`BOr`, `BXor`, `Shl`, `Shr`; the assign statement accepts all five +compound forms and desugars them at parse time. + +- [ ] Extend `ast.ml`: `Not` in `unop`, the five bitwise constructors + in `binop`, with a doc comment recording the Go-rung decision and + pointing at the story's settled-decisions section. +- [ ] Parser rungs: `parse_multiplicative` also accepts `Amp`/`Shl`/ + `Shr`, the additive function also accepts `Pipe`/`Caret`, mapping to + the new constructors. Update the ladder doc comment (~594) to show + the new members of each rung and the two grouping consequences + (`x & mask == 0` groups the AND first; shifts bind tighter than + `+`). The declaration-position use of `Pipe` (union types) parses in + a different grammar and is untouched — confirm by the goldens. +- [ ] `parse_unary` (:908) gains `KwNot` beside `Dash`, producing + `Unary (Not, operand)` — the Lua placement the story settled: + `not a == b` groups `(not a) == b`. +- [ ] Compound assigns at the assign site (:1489): after a place + expression, `PlusEq`/`MinusEq`/`StarEq`/`SlashEq`/`PercentEq` build + the same `Ast.Assign` whose value is a `Binary` of the matching + arithmetic op over a CLONE of the target and the parsed right-hand + side. The clone must freshen every node id (`fresh_id`) — write the + small recursive helper beside `subst_expr`, which is the precedent + for rebuilding expressions. Semantics are therefore exactly the + written-out form (`x += e` ≡ `x = x + e`), including an index + expression evaluating twice — that equivalence is the documented + contract, pinned by a fixture in Task 5. +- [ ] Verify: `just woc-test` unchanged; hand-compile a scratch file + with each new form and confirm the AST dump groups per the ladder + doc. Commit. + +## Task 3 — type checker: Int-only bitwise, Bool-only not, literal shift counts + +**Files:** modify `compiler/src/types.ml` (the `Binary` operand +tables — the And/Or Bool-wiring near :868/:1196 and the arithmetic +table that owns `Add` are the anchors), `compiler/src/diag.ml` if the +new code needs registering. + +**Interfaces produced:** `BAnd`/`BOr`/`BXor`/`Shl`/`Shr` check as +Int × Int → Int; `Not` checks as Bool → Bool; a literal shift count +outside 0..63 is a compile error. + +- [ ] Wire the five bitwise ops into the same checker table `Add` + lives in, Int-only on both sides (no Float twin — nothing like + FADD exists for them, and `numeric_world` stays untouched). The + wrong-operand diagnostic is the existing operator-type family the + corpus fixture `compile-fail/lang-and-non-bool-operand` demonstrates + for and/or — same shape, Int spelled where Bool was. +- [ ] Wire `Not` as Bool → Bool through the same path `And`/`Or` + operands use. +- [ ] Literal shift counts: when a `Shl`/`Shr` right operand is an + integer literal outside 0..63, emit the iteration's one NEW + diagnostic at the operand's position — "shift count N is out of + range 0..63". Take the next free code in the WO-E2 family (grep + diag.ml/types.ml for the current highest; WO-E205 is the last known + at `types.ml:1509`) and register it wherever that family is listed. +- [ ] Verify: `just woc-test`; scratch files confirm each rejection + fires at the right position and each accepted form types as Int/Bool. + Commit. + +## Task 4 — emit and runtime: five opcodes, one trap kind, `.wob` v6 + +**Files:** modify `runtime/src/wob.h` (opcode enum after `WOP_FLE = 41`, +trap enum after `WO_T_FK = 11` at :186, `WOB_VERSION` at :16), +`runtime/src/vm.c` (dispatch table ~1058 and the arithmetic label +block), `runtime/src/loader.c` (the per-opcode validation switch, +~534), `compiler/src/emit.ml` (the `Binary` lowering that maps `Add` +to `WOP_ADD`, and the `Unary` site), `compiler/src/disasm.ml` (opcode +names), plus the runtime's own disassembler if `runtime/src` carries +one (grep for where FADD got its name). + +**Interfaces produced:** `WOP_BAND = 42`, `WOP_BOR = 43`, +`WOP_BXOR = 44`, `WOP_SHL = 45`, `WOP_SHR = 46`; trap kind +`WO_T_SHIFT = 12`; `WOB_VERSION` 5 → 6. + +- [ ] Add the five opcodes to `wob.h` with the house comment style: A + B C register forms on i64; SHL/SHR document the trap contract (count + outside 0..63 traps WO_T_SHIFT) and that SHR is arithmetic + (sign-extending) per the story. Bump `WOB_VERSION` to 6 and extend + its comment with a one-line v6 entry (the v5 comment is the + template). Add `WO_T_SHIFT` to the trap enum with its message wired + wherever the other trap kinds name theirs. +- [ ] VM: five new labels in the computed-goto table (~1058) and + cases in the ISO-switch flavor (both dispatch flavors are gated — + the wovm-test comment in the justfile says why). AND/OR/XOR are + single-expression cases beside ADD; SHL/SHR range-check the count + register first and trap WO_T_SHIFT, then shift (SHR on the signed + value — C's signed right shift is arithmetic on every platform the + runtime supports, but write it via the explicit sign-preserving + idiom the codebase prefers if one exists; check how DIV guards + INT64_MIN for the local style). +- [ ] Loader: the validation switch (~534) accepts the five new + opcodes with three-register operand checking, same arm shape as ADD. +- [ ] Emit: extend the `Binary` lowering table (find where `Add` + becomes `WOP_ADD` — `Ast.` names may be opened, grep for `Concat`'s + lowering) with the five new mappings. Lower `Unary Not` with NO new + opcode: Bool is 0/1, so `not x` is the existing WOP_EQ against a + zero constant (the And/Or comment at `ast.ml:151` records the same + no-new-opcode doctrine for the short-circuit pair). Compound assigns + need nothing here — they died in the parser. +- [ ] Disasm: names for the five opcodes in `compiler/src/disasm.ml` + and the runtime twin if it exists. +- [ ] Verify: `just woc-build && just wovm-build`, then `just + woc-test` and `just wovm-test` (the ASan+UBSan gate, both dispatch + flavors). A scratch program exercising every operator, a + variable-count in-range shift, and a caught out-of-range shift (try/ + catch over the trap) prints the expected values. Commit. + +## Task 5 — corpus, gates, docs: pin everything, close out + +**Files:** create fixtures under `tests/corpus/run/`, +`tests/corpus/compile-fail/`, `tests/corpus/trap/`; modify +`docs/plan/oop-vm/00-wob-format.md` (v6 section), +`compiler/src/CODE-LOGIC.md` and `runtime/src/CODE-LOGIC.md` +(reasoning-under-the-code), the story file (move refine → done, both +frontmatter and folder in the same change), `docs/stories/00-status.md` +(standup entry). + +- [ ] Run fixtures (the `run/arithmetic` fixture is the shape + template): one covering all five bitwise operators including the two + grouping pins (`x & mask == 0`, a shift mixed with `+`); one for + hex/binary/underscore literals proving value-identity with decimal + twins; one for `not` (plain, chained with and/or, and the + `not a == b` grouping pin); one for compound assigns on a let, a + field, and an index place, including the desugar-equivalence pin (an + index expression with a visible side effect running twice, exactly + as the written-out form would). +- [ ] The consumer proof: a fixture computing HMAC's ipad/opad XOR + step in pure `.wo` — `byte_at` over a key Text, `^` with the 0x36 + and 0x5c pad constants — matching the RFC 2104 test-vector + intermediate. This is the story's acceptance criterion that + iteration 34 is demonstrably unblocked. +- [ ] Compile-fail fixtures: `not` on an Int operand; a bitwise + operator on Bool and on Float; a literal shift count of 64 and of a + negative literal (the new diagnostic, both sides of the range). +- [ ] Trap fixture: a variable-count shift receiving 64 at run time + traps WO_T_SHIFT — uncaught surface first, then a try/catch arm + reading the error code, whichever shape `tests/corpus/trap/`'s + existing fixtures use. +- [ ] Dead-token defect closure: confirm the pre-change WO-E101 on + `x += 1` is gone by the compound-assign run fixture existing at all; + note the closure in the story's landing blockquote. +- [ ] Gates, all of them: `just woc-test`, `just wovm-test`, + `just oop-accept`, `just web-app`, `just deps-accept` — every count + at or above its story-recorded level, zero failures. +- [ ] Docs closeout: v6 section in `00-wob-format.md` (five opcodes, + trap kind, version gate — the v5 section is the template); + CODE-LOGIC.md on both sides for the decisions that live in code + (arithmetic SHR, trap-not-mask, `not` as EQ-zero, parse-time + compound-assign desugar and its double-eval contract); story file + moved to `done/` with `status: done` and a landing blockquote in the + story-15/16 voice; standup entry in `00-status.md` answering the six + questions (reference project: `.dev/reference/go`). +- [ ] Commit. + +## Self-review notes + +- Story coverage: goals 1-5 map to Tasks 1-4; every acceptance + criterion has a Task-5 fixture or gate; the settled decisions are + restated at their implementation sites so no executor re-litigates + them. +- Type consistency: token names (Amp/Caret/Shl/Shr/StarEq/SlashEq/ + PercentEq/KwNot), AST names (Not, BAnd/BOr/BXor/Shl/Shr), opcode + names/ids (WOP_BAND..WOP_SHR = 42..46), and the trap kind + (WO_T_SHIFT = 12) are spelled identically in every task that touches + them. +- Known allowance: Task 3's diagnostic code is "next free in the + WO-E2 family" rather than a fixed number — the family's registry is + the source of truth and hard-coding here would rot.