docs: story 36 + plan — operator parity (not, bitwise, hex literals, compound assigns)

- story: gap survey vs Go reference; forks settled (arithmetic >>, trap
  on out-of-range shift count, Lua-placement not)
- plan: 5 tasks, lexer -> parser -> checker -> emit/VM (.wob v6) -> closeout
- latent defect recorded: += / -= lex but never parse (dead tokens)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
shoney.arickathil 2026-08-22 21:18:33 +02:00
parent b148eb98a6
commit 0b87c6c5f3
2 changed files with 416 additions and 0 deletions

View file

@ -0,0 +1,155 @@
---
iteration: "36"
status: refine
---
# Iteration 36 — operator parity: `not`, bitwise, hex literals, compound assigns
> Format: fiberloom `product/story-iteration-template`. Part of
> [Story — one language, one runtime, one database, one binary](../00-story.md).
>
> **Inserted 2026-08-22, forks settled the same day** (decisions below,
> story-19 discipline). Driver: a gap survey of the operator surface
> against the Go reference checkout (`.dev/reference/go` —
> `src/go/token/token.go`, spec §Arithmetic operators) found four holes
> that every mainstream language covers. The survey also found a latent
> defect: `+=`/`-=` are lexed (`lexer.ml:541/548`, tokens at
> `token.ml:149-150`) but no parser rule consumes them — dead tokens
> since the haxe-parity control-surface task, zero uses in the corpus,
> so `x += 1` today dies as a generic WO-E101 instead of working or
> being honestly absent.
>
> The driving consumer is iteration 34 (crypto builtins): HMAC's
> ipad/opad step is a byte-wise XOR, and the doctrine question recorded
> there — C-side builtin vs pure-`.wo` — is unanswerable while the
> language cannot spell XOR at all. Secondary consumer: `http/auth.wo`'s
> pure-`.wo` base64 does with division/modulo what shifts and masks say
> directly.
>
> **Plan:** [`2026-08-22-operator-parity.md`](../../../superpowers/plans/2026-08-22-operator-parity.md)
> (5 tasks: lexer/tokens; AST/parser incl. compound-assign desugar;
> type checker incl. literal shift-count rejection; emit + VM opcodes
> `.wob` v6; corpus/gates/docs closeout).
## Why this iteration exists
The expression grammar (`parser.ml` ~594, `token.ml:134-150`) stops at:
arithmetic `+ - * / %`, concat `..`, comparison `== != < <= > >=`,
logical `and`/`or`, unary minus. Four common gaps, in priority order:
1. **No boolean negation.** No `not`, no `!`, no unary NOT node in the
AST. The only spelling is `x == false` — every caller pays, every
day. Highest value for smallest surface.
2. **No bitwise operators.** No `&`, `|`, `^`, `<<`, `>>` in any form.
Blocks iteration 34's HMAC honestly; base64/multipart boundary code
emulates masks with arithmetic.
3. **No hex/binary integer literals.** The lexer scans decimal digit
runs only (`lexer.ml:386` on). Bitwise code without `0xFF` masks is
write-only.
4. **Compound assigns half-promised.** `+=`/`-=` lex but never parse
(the latent defect above); `*=` `/=` `%=` do not exist at all.
Deliberately NOT gaps (doctrine, not omission): `&&`/`||`/`!` symbol
forms (words won — KwAnd/KwOr precedent), ternary `?:` (switch is
already an expression, `parser.ml:1181`), `++`/`--`, function values,
inheritance-family keywords (WO-E105 rejects them by name).
## Goals
- **`not` keyword**, unary boolean, sitting with unary minus in the
grammar (settled below: Lua placement, binds tighter than comparison).
Bool-only operand, diagnosed like every other type error.
- **Five bitwise operators on Int only**: `&` AND, `|` OR, `^` XOR,
`<<` left shift, `>>` right shift. Precedence copies Go's C-trap fix
(`token.go:266-280`): `&`/`<<`/`>>` at the multiplicative level,
`|`/`^` at the additive level — both ABOVE comparison, so
`x & mask == 0` groups the AND first. Complement is spelled
`-1 ^ x` (single signed 64-bit Int makes Go's mask rule collapse to
exactly this); no `~`, no unary `^`, no `&^`.
- **Hex and binary literals**: `0x` and `0b` prefixes on Int, plus `_`
digit separators in all integer forms. Decimal lexing stays
byte-identical for every existing fixture (the iteration-19
discipline).
- **Compound assigns wired**: `+=` `-=` parse into the existing
assignment statement (resurrecting the dead tokens), `*=` `/=` `%=`
join them. Statement-level sugar over the existing place logic —
no new AST evaluation semantics.
- **Runtime**: new i64 register opcodes beside `WOP_ADD..WOP_NEG`
(`runtime/src/wob.h:194`) for AND/OR/XOR/SHL/SHR (+NOT if the emitter
wants it); `.wob` version bump, loader validation, disasm coverage —
the v5 (Float/Bytes) change is the template.
## Settled decisions (2026-08-22)
1. **`>>` is arithmetic (sign-extending).** Go's own choice for signed
integers, and writeonce has exactly one Int, signed 64-bit. For the
driving consumers the fork is moot anyway — crypto/base64 shift
byte-range values whose sign bit is never set, where arithmetic and
logical are bit-identical. No logical-shift operator ships; a caller
who wants zero-fill masks first.
2. **Out-of-range shift count traps, like DIV0.** Valid counts are
0..63; a negative or ≥64 count at run time is a named trap on the
same machinery as `WOP_DIV`'s DIV0 (`wob.h:197`) — the established
Int honesty precedent (story 19 kept the trap deliberately). NOT
Go's saturate-to-0/-1 (spec surface serving generic-width code
writeonce doesn't have) and NOT hardware masking (x86's count%64
makes `x << 64 == x`, the classic silent wat). Real code shifts by
literals: when the count is a literal, the compiler rejects it at
emit time and the trap never runs.
3. **`not` sits at the unary level, beside minus (Lua placement).**
`not a == b` parses `(not a) == b`. The grammar's ordering is
already anchored to Lua by name (`parser.ml` ~601 "This ordering
matches Lua's"), and joining `parse_unary` adds zero new precedence
rungs. Python's looser placement reads closer to English, but it
buys a new rung to prevent a misparse that Bool-only typing already
converts into a compile error in every mixed-type case; the one
silent case (Bool compared to Bool through `not`) gets a pinned
fixture so the choice stays visible. `if not done` and `while not
empty()` — the actual daily uses — read identically under both.
## Acceptance Criteria
- **Given** the corpus, **when** fixtures land for `not` (plain,
chained with and/or, non-Bool operand diagnosed), each bitwise
operator, precedence pins (`x & mask == 0` and one shift-vs-additive
case), hex/binary/underscore literals (value-identical to decimal
twins), out-of-range shift (literal count rejected at compile time,
variable count trapping at run time), the `not a == b` grouping pin,
and all five compound assigns on let/field/index places, **then**
corpus passes with zero regressions on every pre-existing fixture.
- **Given** iteration 34's ipad/opad step written in pure `.wo` with
`^` over `byte_at` values, **when** it runs against an RFC 2104 test
vector's intermediate, **then** the bytes match — the consumer that
motivated the iteration is demonstrably unblocked.
- **Given** the existing gates (`just corpus`-equivalent, `just
web-app`, `just deps-accept`, `test_wal`), **when** the iteration
lands, **then** all pass unchanged — the `.wob` bump breaks no
replay/loader path.
- **Given** `x += 1` in a fixture today-vs-after, **when** compiled,
**then** the before is the recorded WO-E101 and the after executes —
the dead-token defect is provably closed, not papered over.
## Out Of Scope
Unary complement operator (spell `-1 ^ x`), Go's `&^` AND-NOT, bitwise
compound assigns (`&=` family), `++`/`--`, ternary, symbol forms
`&&`/`||`/`!`, bitwise on Float/Bytes/Bool, overflow-checked arithmetic
changes, octal literals.
## Proposed Solution
Lexer: extend the operator switch (the `+`/`+=` two-char pattern at
`lexer.ml:541-548` is the template) for the five operators and the
three new compound assigns; extend the digit scanner with `0x`/`0b`
prefix branches and `_` skipping, leaving the bare-decimal path
untouched. Parser: two new precedence rungs threaded into the existing
recursive-descent chain exactly where Go's table says; `not` joins
`parse_unary`; compound assigns join the statement that already
matches `Token.Eq` (`parser.ml:510`). Types: all five operators and
`not` are `Int -> Int -> Int` / `Bool -> Bool` in the same checker
table that owns `+`. Emit/runtime: new opcodes in `wob.h`'s i64 block,
`WOB_VERSION` bump, interpreter cases beside the existing wrapping
arithmetic, disasm names. The forks are settled above, so no separate
spec is owed (the story-19 deviation precedent: format decisions
recorded normatively where they land, reasoning in CODE-LOGIC.md); the
plan follows once this story is approved on the board.

View file

@ -0,0 +1,261 @@
# Iteration 36 — operator parity (`not`, bitwise, hex literals, compound assigns): implementation plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or
> superpowers:executing-plans to implement this plan task-by-task.
> Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** the language gains boolean `not`, the five Int bitwise
operators (`&` `|` `^` `<<` `>>`), hex/binary/underscore integer
literals, and working compound assigns (`+=` `-=` `*=` `/=` `%=`) —
full stack, `.wob` v6.
**Architecture:** front-to-back in dependency order — tokens/lexer,
then AST/parser, then the type checker, then emit + VM opcodes, then
the corpus/gates closeout. Every semantic decision is already settled
in the story ("Settled decisions", 2026-08-22): `>>` arithmetic,
out-of-range shift count traps (literal counts rejected at compile
time), `not` at the unary level beside minus.
**Tech stack:** OCaml compiler (`compiler/src/`), C runtime/VM
(`runtime/src/`), dune + just gates.
**Spec:** the story IS the spec —
`docs/stories/language-runtime-database/refine/36-operator-parity.md`
(story-19 deviation precedent: decisions recorded normatively there,
reasoning-under-the-code lands in CODE-LOGIC.md as part of this plan).
## Global constraints
- Plans in this repo carry no code blocks — each step names the exact
file/anchor and describes the change; the executor writes the code.
- Every pre-existing fixture must lex, parse, and run byte-identically
(the iteration-19 discipline). Any golden that changes is a defect in
the change, not a bless-and-move-on.
- Precedence copies Go (`.dev/reference/go/src/go/token/token.go:266`):
`&` `<<` `>>` join the multiplicative rung, `|` `^` join the additive
rung — no new rungs, both above comparison.
- Bitwise is Int-only. No Float, Bool, Text, Bytes operands — the
existing operand-type diagnostics reject them; `numeric_world`
(`types.ml:193`) is not touched.
- Run gates via just recipes only (`just woc-test`, `just wovm-test`,
`just oop-accept`, `just web-app`, `just deps-accept`).
- Branch off master first (never commit on master); bullet-style commit
messages, ≤25 lines, no push.
---
## Task 1 — lexer and tokens: the new surface exists as tokens
**Files:** modify `compiler/src/token.ml` (kind list ~134-150),
`compiler/src/lexer.ml` (keyword table ~131-147, operator switch
~500-570, digit scanner ~386-430), `compiler/src/dump.ml` (kind labels
~103).
**Interfaces produced:** tokens `Amp`, `Caret`, `Shl`, `Shr`,
`StarEq`, `SlashEq`, `PercentEq`, keyword `KwNot`; hex (`0x`), binary
(`0b`), and underscore-separated integer literals arriving as ordinary
`Token.Int`.
- [ ] Add the seven operator tokens and `KwNot` to `token.ml`'s kind
list, each with the one-line comment style the file already uses;
add `not` to the lexer keyword table beside `and`/`or`
(`lexer.ml:144-145`) — grep the corpus and samples first to confirm
`not` is used nowhere as an identifier (the discipline every keyword
above it followed).
- [ ] Extend the operator switch: new `&` and `^` cases (single-char,
today they fall to the error branch); `*`, `/`, `%` gain a peek for
`=` (the `+`/`+=` two-char pattern at `lexer.ml:543-549` is the
template); `<` gains a peek for a second `<` (order: `=` first so
`<=` stays `LtEq`, then `<`), `>` symmetrically. No `<<=`/`>>=` —
bitwise compound assigns are out of scope per the story.
- [ ] Extend the digit scanner: after an initial `0`, a peek for `x`
or `b` switches to hex/binary accumulation (case-insensitive hex
digits; at least one digit required, else the existing malformed-
number path); `_` is skipped between digits in every integer form
(leading/trailing/doubled `_` falls out as an error naturally).
Accumulation wraps in i64 like the VM's own arithmetic (the ADD
doctrine, `wob.h:194`). The bare-decimal and float paths stay
byte-identical — the iteration-19 comment at `lexer.ml:387` explains
exactly which branch keeps them safe.
- [ ] Add dump labels for the eight new kinds beside PLUSEQ/MINUSEQ
(`dump.ml:103`).
- [ ] Verify: `just woc-test` — every existing golden unchanged (new
tokens are unreachable from old source). Commit.
## Task 2 — AST and parser: the grammar accepts what the tokens spell
**Files:** modify `compiler/src/ast.ml` (unop at :149, binop at :158),
`compiler/src/parser.ml` (ladder doc ~594, additive/multiplicative
~860-905, unary at :908, assign statement at :1489, an id-freshening
clone helper beside `subst_expr` ~2068).
**Interfaces produced:** `unop` gains `Not`; `binop` gains `BAnd`,
`BOr`, `BXor`, `Shl`, `Shr`; the assign statement accepts all five
compound forms and desugars them at parse time.
- [ ] Extend `ast.ml`: `Not` in `unop`, the five bitwise constructors
in `binop`, with a doc comment recording the Go-rung decision and
pointing at the story's settled-decisions section.
- [ ] Parser rungs: `parse_multiplicative` also accepts `Amp`/`Shl`/
`Shr`, the additive function also accepts `Pipe`/`Caret`, mapping to
the new constructors. Update the ladder doc comment (~594) to show
the new members of each rung and the two grouping consequences
(`x & mask == 0` groups the AND first; shifts bind tighter than
`+`). The declaration-position use of `Pipe` (union types) parses in
a different grammar and is untouched — confirm by the goldens.
- [ ] `parse_unary` (:908) gains `KwNot` beside `Dash`, producing
`Unary (Not, operand)` — the Lua placement the story settled:
`not a == b` groups `(not a) == b`.
- [ ] Compound assigns at the assign site (:1489): after a place
expression, `PlusEq`/`MinusEq`/`StarEq`/`SlashEq`/`PercentEq` build
the same `Ast.Assign` whose value is a `Binary` of the matching
arithmetic op over a CLONE of the target and the parsed right-hand
side. The clone must freshen every node id (`fresh_id`) — write the
small recursive helper beside `subst_expr`, which is the precedent
for rebuilding expressions. Semantics are therefore exactly the
written-out form (`x += e` ≡ `x = x + e`), including an index
expression evaluating twice — that equivalence is the documented
contract, pinned by a fixture in Task 5.
- [ ] Verify: `just woc-test` unchanged; hand-compile a scratch file
with each new form and confirm the AST dump groups per the ladder
doc. Commit.
## Task 3 — type checker: Int-only bitwise, Bool-only not, literal shift counts
**Files:** modify `compiler/src/types.ml` (the `Binary` operand
tables — the And/Or Bool-wiring near :868/:1196 and the arithmetic
table that owns `Add` are the anchors), `compiler/src/diag.ml` if the
new code needs registering.
**Interfaces produced:** `BAnd`/`BOr`/`BXor`/`Shl`/`Shr` check as
Int × Int → Int; `Not` checks as Bool → Bool; a literal shift count
outside 0..63 is a compile error.
- [ ] Wire the five bitwise ops into the same checker table `Add`
lives in, Int-only on both sides (no Float twin — nothing like
FADD exists for them, and `numeric_world` stays untouched). The
wrong-operand diagnostic is the existing operator-type family the
corpus fixture `compile-fail/lang-and-non-bool-operand` demonstrates
for and/or — same shape, Int spelled where Bool was.
- [ ] Wire `Not` as Bool → Bool through the same path `And`/`Or`
operands use.
- [ ] Literal shift counts: when a `Shl`/`Shr` right operand is an
integer literal outside 0..63, emit the iteration's one NEW
diagnostic at the operand's position — "shift count N is out of
range 0..63". Take the next free code in the WO-E2 family (grep
diag.ml/types.ml for the current highest; WO-E205 is the last known
at `types.ml:1509`) and register it wherever that family is listed.
- [ ] Verify: `just woc-test`; scratch files confirm each rejection
fires at the right position and each accepted form types as Int/Bool.
Commit.
## Task 4 — emit and runtime: five opcodes, one trap kind, `.wob` v6
**Files:** modify `runtime/src/wob.h` (opcode enum after `WOP_FLE = 41`,
trap enum after `WO_T_FK = 11` at :186, `WOB_VERSION` at :16),
`runtime/src/vm.c` (dispatch table ~1058 and the arithmetic label
block), `runtime/src/loader.c` (the per-opcode validation switch,
~534), `compiler/src/emit.ml` (the `Binary` lowering that maps `Add`
to `WOP_ADD`, and the `Unary` site), `compiler/src/disasm.ml` (opcode
names), plus the runtime's own disassembler if `runtime/src` carries
one (grep for where FADD got its name).
**Interfaces produced:** `WOP_BAND = 42`, `WOP_BOR = 43`,
`WOP_BXOR = 44`, `WOP_SHL = 45`, `WOP_SHR = 46`; trap kind
`WO_T_SHIFT = 12`; `WOB_VERSION` 5 → 6.
- [ ] Add the five opcodes to `wob.h` with the house comment style: A
B C register forms on i64; SHL/SHR document the trap contract (count
outside 0..63 traps WO_T_SHIFT) and that SHR is arithmetic
(sign-extending) per the story. Bump `WOB_VERSION` to 6 and extend
its comment with a one-line v6 entry (the v5 comment is the
template). Add `WO_T_SHIFT` to the trap enum with its message wired
wherever the other trap kinds name theirs.
- [ ] VM: five new labels in the computed-goto table (~1058) and
cases in the ISO-switch flavor (both dispatch flavors are gated —
the wovm-test comment in the justfile says why). AND/OR/XOR are
single-expression cases beside ADD; SHL/SHR range-check the count
register first and trap WO_T_SHIFT, then shift (SHR on the signed
value — C's signed right shift is arithmetic on every platform the
runtime supports, but write it via the explicit sign-preserving
idiom the codebase prefers if one exists; check how DIV guards
INT64_MIN for the local style).
- [ ] Loader: the validation switch (~534) accepts the five new
opcodes with three-register operand checking, same arm shape as ADD.
- [ ] Emit: extend the `Binary` lowering table (find where `Add`
becomes `WOP_ADD` — `Ast.` names may be opened, grep for `Concat`'s
lowering) with the five new mappings. Lower `Unary Not` with NO new
opcode: Bool is 0/1, so `not x` is the existing WOP_EQ against a
zero constant (the And/Or comment at `ast.ml:151` records the same
no-new-opcode doctrine for the short-circuit pair). Compound assigns
need nothing here — they died in the parser.
- [ ] Disasm: names for the five opcodes in `compiler/src/disasm.ml`
and the runtime twin if it exists.
- [ ] Verify: `just woc-build && just wovm-build`, then `just
woc-test` and `just wovm-test` (the ASan+UBSan gate, both dispatch
flavors). A scratch program exercising every operator, a
variable-count in-range shift, and a caught out-of-range shift (try/
catch over the trap) prints the expected values. Commit.
## Task 5 — corpus, gates, docs: pin everything, close out
**Files:** create fixtures under `tests/corpus/run/`,
`tests/corpus/compile-fail/`, `tests/corpus/trap/`; modify
`docs/plan/oop-vm/00-wob-format.md` (v6 section),
`compiler/src/CODE-LOGIC.md` and `runtime/src/CODE-LOGIC.md`
(reasoning-under-the-code), the story file (move refine → done, both
frontmatter and folder in the same change), `docs/stories/00-status.md`
(standup entry).
- [ ] Run fixtures (the `run/arithmetic` fixture is the shape
template): one covering all five bitwise operators including the two
grouping pins (`x & mask == 0`, a shift mixed with `+`); one for
hex/binary/underscore literals proving value-identity with decimal
twins; one for `not` (plain, chained with and/or, and the
`not a == b` grouping pin); one for compound assigns on a let, a
field, and an index place, including the desugar-equivalence pin (an
index expression with a visible side effect running twice, exactly
as the written-out form would).
- [ ] The consumer proof: a fixture computing HMAC's ipad/opad XOR
step in pure `.wo` — `byte_at` over a key Text, `^` with the 0x36
and 0x5c pad constants — matching the RFC 2104 test-vector
intermediate. This is the story's acceptance criterion that
iteration 34 is demonstrably unblocked.
- [ ] Compile-fail fixtures: `not` on an Int operand; a bitwise
operator on Bool and on Float; a literal shift count of 64 and of a
negative literal (the new diagnostic, both sides of the range).
- [ ] Trap fixture: a variable-count shift receiving 64 at run time
traps WO_T_SHIFT — uncaught surface first, then a try/catch arm
reading the error code, whichever shape `tests/corpus/trap/`'s
existing fixtures use.
- [ ] Dead-token defect closure: confirm the pre-change WO-E101 on
`x += 1` is gone by the compound-assign run fixture existing at all;
note the closure in the story's landing blockquote.
- [ ] Gates, all of them: `just woc-test`, `just wovm-test`,
`just oop-accept`, `just web-app`, `just deps-accept` — every count
at or above its story-recorded level, zero failures.
- [ ] Docs closeout: v6 section in `00-wob-format.md` (five opcodes,
trap kind, version gate — the v5 section is the template);
CODE-LOGIC.md on both sides for the decisions that live in code
(arithmetic SHR, trap-not-mask, `not` as EQ-zero, parse-time
compound-assign desugar and its double-eval contract); story file
moved to `done/` with `status: done` and a landing blockquote in the
story-15/16 voice; standup entry in `00-status.md` answering the six
questions (reference project: `.dev/reference/go`).
- [ ] Commit.
## Self-review notes
- Story coverage: goals 1-5 map to Tasks 1-4; every acceptance
criterion has a Task-5 fixture or gate; the settled decisions are
restated at their implementation sites so no executor re-litigates
them.
- Type consistency: token names (Amp/Caret/Shl/Shr/StarEq/SlashEq/
PercentEq/KwNot), AST names (Not, BAnd/BOr/BXor/Shl/Shr), opcode
names/ids (WOP_BAND..WOP_SHR = 42..46), and the trap kind
(WO_T_SHIFT = 12) are spelled identically in every task that touches
them.
- Known allowance: Task 3's diagnostic code is "next free in the
WO-E2 family" rather than a fixed number — the family's registry is
the source of truth and hard-coding here would rot.