From 7f9332f5d2a74fe6c82721674f35e2c61fb045f0 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Sat, 15 Aug 2026 02:00:01 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20iteration=209b=20spec=20+=20plan=20?= =?UTF-8?q?=E2=80=94=20@table,=20relations,=20query=20(employee)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - spec settles 9b's three forks: SQL/Cypher layer superseded as the program surface (design history + wo-db engine-semantics reference); comprehension syntax desugared at compile time (no function values); System.Linq = operator vocabulary + edge cases, PostgreSQL = execution + integrity vocabulary (both references surveyed 2026-08-15) - aggregate semantics normative: count/sum total 0 on empty, avg/min/max are ?T with nil (empty is data, not a fault); nil skipped; sum wraps like language arithmetic; GroupBy lowers as group-and-reduce (AggregateBy shape), transition/finalize ABI from nodeAgg - relations: ref = FK with direct-index-probe check (nil passes, unchanged-key skips), backlink = secondary-index scan, delete is restrict-only; nil never joins, nil is a legal group key - lowering: the compiler is the planner — queries become bytecode loops over cursor/group builtins, longest-prefix index selection, no plan tree, no SQL text in the image (disassembly-provable) - plan: 6 tasks gated by a new docs/examples/employee sample (Department/Employee, @unique, composite index, ref/backlink, GroupBy report mode) with its own acceptance script + crash step; blocked on iteration 9's engine plan - story 09b + status board updated; 02-wo-language.md carries the supersession note Co-Authored-By: Claude Opus 5 (1M context) --- docs/00-status.md | 4 +- .../2026-08-15-employee-relations-query.md | 200 ++++++++++++ docs/runtime/database/02-wo-language.md | 8 + .../09b-table-relations-query.md | 27 +- ...2026-08-15-table-relations-query-design.md | 307 ++++++++++++++++++ 5 files changed, 537 insertions(+), 9 deletions(-) create mode 100644 docs/plan/compiler/2026-08-15-employee-relations-query.md create mode 100644 docs/superpowers/specs/2026-08-15-table-relations-query-design.md diff --git a/docs/00-status.md b/docs/00-status.md index 49c7d2d..f4906f9 100644 --- a/docs/00-status.md +++ b/docs/00-status.md @@ -116,7 +116,7 @@ that sequences its tasks. Read one, approve, then the next starts. | 7b | [Inferred GC + mark-sweep](stories/language-runtime-database/07b-inferred-gc-mark-sweep.md) | ⏸ off the workload's path (no `@gc`) | | 8 | [Shard-actor runtime](stories/language-runtime-database/08-shard-actor-runtime.md) | ⬜ | | 9 | [Database engine](stories/language-runtime-database/09-database-engine.md) | ⬜ | -| 9b | [`@table`, relations, query](stories/language-runtime-database/09b-table-relations-query.md) | ⬜ needs a spec first | +| 9b | [`@table`, relations, query](stories/language-runtime-database/09b-table-relations-query.md) | ⬜ spec + plan ready (2026-08-15) | | 10 | [HTTP service layer](stories/language-runtime-database/10-http-service.md) | ⬜ | Hold | | 11 | [Fibers](stories/language-runtime-database/11-fibers.md) | ⬜ | Hold | | 12 | [Blue-green deploy](stories/language-runtime-database/12-blue-green-deploy.md) | ⬜ | Hold | @@ -322,7 +322,7 @@ Ecommerce sample (verified 2026-06-13): `api.rest` 17/17 expected statuses pass. | 7b | Inferred GC + incremental mark-sweep — `@gc` removed, GC-ness inferred, RC retired | [spec](superpowers/specs/2026-08-11-inferred-gc-mark-sweep-design.md) — plan to be written | | 8 | Shard-actor runtime | [plan 4](superpowers/plans/2026-08-01-shard-actor-vm-runtime.md) | | 9 | Database engine binding | [plan 5](superpowers/plans/2026-08-01-db-engine-binding.md) | -| 9b | `@table` + relations + language-integrated query | **no spec yet** — three open forks recorded in the iteration; brainstorm before planning | +| 9b | `@table` + relations + language-integrated query — comprehension queries, `ref`/`backlink` navigation, GroupBy aggregates; acceptance: new `docs/examples/employee` sample | [spec](superpowers/specs/2026-08-15-table-relations-query-design.md) · [plan](plan/compiler/2026-08-15-employee-relations-query.md) | | 10 | HTTP service layer | [plan 6](superpowers/plans/2026-08-01-http-service-layer.md) | | 11 | Fibers | vision §3, [blue-green exploration](plan/exploration/blue-green-vm/00-vision.md) | | 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9–10 | diff --git a/docs/plan/compiler/2026-08-15-employee-relations-query.md b/docs/plan/compiler/2026-08-15-employee-relations-query.md new file mode 100644 index 0000000..e1579b5 --- /dev/null +++ b/docs/plan/compiler/2026-08-15-employee-relations-query.md @@ -0,0 +1,200 @@ +# `@table`, Relations, Query — Implementation Plan (employee sample) + +> **Status: ⬜ pending** (story iteration 9b) — blocked on iteration 9's engine +> plan ([`2026-08-01-db-engine-binding.md`](../../superpowers/plans/2026-08-01-db-engine-binding.md)): +> Tasks 3–6 below consume its row storage, WAL, indexes and select subset. +> Board: [00-status.md](../../00-status.md) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> +> **Style rule (user convention):** concept, reason, and required behavior in words only; the executor writes the code. + +**Spec:** [`docs/superpowers/specs/2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md) (normative: the settled forks, the clause grammar, the aggregate semantics table, the lowering model), amended by the systems-track spec's program-mode contract for the sample's CLI. + +**Goal:** `@table` classes queried **in the language**: comprehension queries +with `where`/`join`/`group`/`order`/`take`/`select`, typed `ref`/`backlink` +navigation with restrict integrity, and the LINQ aggregate vocabulary +(`count`/`sum`/`avg`/`min`/`max`, `GroupBy` as group-and-reduce) — all lowered +to bytecode loops over engine cursor builtins, proven by a new +`docs/examples/employee` sample whose acceptance script is the gate. + +**Architecture:** compiler front (`compiler/src/{lexer,parser,ast,types,owner,emit}.ml`) +for the query surface; runtime (`runtime/src/{table,db,builtin}.c` from +iteration 9, plus a new `query.c` for cursors and group hashing). The compiler +is the planner: index selection happens at lowering, the VM never sees a plan +tree. Reference semantics: System.Linq for operator meaning, PostgreSQL's +nodeAgg/ri_triggers for execution and integrity vocabulary (both surveyed in +the spec, fork 3). + +**Tech Stack:** OCaml stdlib (compiler), C11 libc (runtime). No new opcodes; +new builtin ids appended to the format doc. + +## Global Constraints + +- **The sample is the test.** `docs/examples/employee` plus its acceptance + script is 9b's gate; the only corpus additions are the db-corpus fixtures + iteration 9 already plans. `just oop-e2e`, `just woc-test`, `just wovm-test` + stay green as regression after every task. +- **Index doctrine** (iteration 9, verbatim): secondary indexes are maintained + only through the engine's row choke points; FK checks and backlink reads are + index probes, never storage walks. +- **No SQL text anywhere** — the 9b acceptance's disassembly criterion. +- **No function values.** Every predicate/projection is expression syntax; a + query value is never deferred or passed around. +- Plans/specs in words; commits local only, never push; docs under `docs/`; + CODE-LOGIC.md updated beside changed code. + +## File Structure + +``` +compiler/src/lexer.ml parser.ml ast.ml query-expression grammar, clause AST (Task 1) +compiler/src/types.ml range/group scopes, navigation, projection synthesis (Task 2) +compiler/src/owner.ml emit.ml query ownership + lowering to cursor builtins (Task 5) +runtime/src/query.c query.h cursors, group hash, transition/finalize aggregates (Tasks 3, 4) +runtime/src/db.c table.c FK restrict checks at the row choke points (Task 3) +docs/examples/employee/ the acceptance workload (Task 6) +scripts/employee-accept.sh the gate (Task 6) +docs/plan/oop-vm/00-wob-format.md appended builtin ids (Tasks 3-5) +``` + +--- + +### Task 1: Query expressions parse + +**Concept & reason:** the comprehension grammar from spec section 3 — +`from`/`where`/`join`/`group…into`/`order by`/`take`/`skip`/`select`, clause +order fixed, aggregates as clause functions — becomes lexer keywords (contextual, +so `from`/`group`/`order` stay legal identifiers outside a query), a clause AST, +and parse diagnostics in a new WO-E5xx range (clause out of order, missing +`select`, aggregate named outside a query). Fixed clause order is a parse rule, +not a type rule, so the error lands on the exact token. + +- [ ] Grammar and AST for every clause; contextual keywords verified against + the existing samples (no `.wo` file in the tree breaks). +- [ ] WO-E5xx diagnostics with golden coverage in the existing `woc-test` + suite (dump-ast goldens for well-formed queries, diagnostic goldens for + each malformed shape). +- [ ] Gates green; commit locally. + +### Task 2: Queries typecheck + +**Concept & reason:** the surface's whole promise is compile-time checking. +`from e in Employee` opens a scope where `e`'s fields are the class's; `ref` +navigation substitutes the target class's field set (`e.dept.name`); +`backlink` reads type as `multi` of the source class; `group … into g` closes +the range scope and opens the group scope, where `g.key` has the key's type +and `g.f` is legal **only** inside an aggregate call; `select { … }` +synthesizes an anonymous record type so later use of a dropped column is a +compile error. Aggregate result types follow the spec's table exactly — +`count` is Int, `sum` Int, `avg`/`min`/`max` are `?T` because an empty group +is data. Unknown table, unknown column (naming the class), navigation through +a non-`ref`, bare group-member column, join sides not separable: each is its +own WO-E5xx with a golden. + +- [ ] Scope machinery for range/group variables; navigation typing both + directions; projection record synthesis interned like other class + shapes. +- [ ] Aggregate typing per the spec table; `?T` results force the existing + nil-handling style at use sites. +- [ ] Diagnostic goldens for every error class named above; gates green; + commit locally. + +### Task 3: Cursors and integrity in the engine + +**Concept & reason:** the runtime side queries need, built on iteration 9's +storage: a table-scan cursor (open by class id, advance, borrowed row view), +a primary-index point read (`ref` navigation), a secondary-index cursor with +longest-prefix probe (backlink reads, indexed `where`), all as builtins +appended to the format doc. Integrity lands at the row choke points where the +indexes already live: inserting/updating a non-nil `ref` probes the referenced +primary index and traps on a miss (foreign-key violation, sibling of the +unique trap); deleting a row still referenced traps (restrict) via the same +secondary index a `backlink` requires — one probe, no new structure. Nil +`ref` never probes (the MATCH SIMPLE rule); an update that leaves the key +unchanged skips the check (both from the PostgreSQL RI survey, mechanism +discarded — a check is an index probe, never query text). + +- [ ] Cursor builtins + borrowed-row-view lifetime rules written into the + binding doc (a row view never escapes the loop that opened the cursor — + the ownership pass enforces it, mirror of the container-read borrow). +- [ ] FK trap + restrict trap wired through the row API; a debug-build probe + counter exposed for Task 5's index-selection proof. +- [ ] db-corpus fixtures from iteration 9's plan extended with one FK-violation + and one restrict fixture (trap-code exact); ASan green; commit locally. + +### Task 4: Group hash + aggregate execution + +**Concept & reason:** the `AggregateBy` shape from the spec — group-and-reduce +in one pass, no group objects. A group hash table builtin set: create (keyed +by the group key's kind, nil legal), upsert-advance (locate-or-create the +group, advance each aggregate's transition slot), drain (iterate groups, +finalize, hand key + finals to the compiled projection loop). Transition/ +finalize is PostgreSQL's split: `avg` carries sum+count in its transition +state and divides at finalize; the "no row seen yet" state is distinct from +"transition value is nil" so nil-skipping aggregates need no first-row special +case. Aggregate semantics are the spec's table: empty ⇒ nil (or 0 for +count/sum), nil elements skipped, sum wraps, avg truncates. + +- [ ] Builtins implemented; the input projected to only the columns the + aggregates read before hashing (the nodeAgg memory lesson). +- [ ] Fixture-level verification through iteration 9's db corpus (one grouped + query, exact output) plus the sample in Task 6; ASan green; commit + locally. + +### Task 5: Lowering — the compiler is the planner + +**Concept & reason:** a query desugars to the bytecode loops the language +already has. Scan or probe chosen at compile time: leading `where` equality +conjuncts matched against declared indexes longest-prefix-first, probe emitted +on a hit, scan otherwise — and the choice is **demonstrated** via Task 3's +probe counter in the acceptance, not assumed. `join` builds a transient hash +on the inner side and probes with the outer (nil never inserted, never +probes). `group` emits the two-phase hash-aggregation loop from Task 4. +`order by` materializes and stable-sorts (original-index tiebreak — the LINQ +stability guarantee); `take`/`skip` slice the result. Ownership: row views +stay loop-bound borrows; everything `select` emits is copied/built at the +boundary under the established Text-copy and fresh-value rules — queries add +no new ownership classes, and the ownership pass's existing drop machinery +covers the query's temporaries because the lowering IS ordinary loops. + +- [ ] Desugar + lowering for every clause; disassembly of the sample's report + mode shows loops and builtins, no plan tree, no text. +- [ ] Index selection proven: the acceptance asserts probe-counter deltas for + the indexed `staff ` path versus a full-scan query. +- [ ] `oop-e2e`, `woc-test`, ASan-corpus gates green; commit locally. + +### Task 6: The employee sample is the acceptance + +**Concept & reason:** spec section 6 verbatim — `docs/examples/employee` with +`Department`/`Employee` (`@table`, `@unique` name, composite `[dept, salary]` +index, `ref`/`backlink` pair), wo.toml manifest so `woc .` builds it, a `just` +module beside it (the log-watcher convention), and `scripts/employee-accept.sh` +as the gate: `seed` (insert + WAL + duplicate-department trap on the second +run), `report` (headcount/avg/min/max by department, total payroll, ordered by +average salary — byte-exact lines), `staff` (both navigation directions + +index-probe proof), `raise` (update through a query; missing department ⇒ the +empty-query nil path), the restrict-trap demo, and a kill -9 between `seed` +and `report` proving replay on this workload. + +- [ ] Sample written; `woc docs/examples/employee` compiles with zero + diagnostics; the binary's modes run. +- [ ] `scripts/employee-accept.sh` + `just employee` module: every mode + checked with exact expectations, trap codes asserted, crash step + included; soak-style RSS/fd sampling reused from the log-watcher + script's pattern for the `report` loop. +- [ ] ASan run of the full acceptance: zero leaks (the log-watcher bar). +- [ ] Docs: README beside the sample, CODE-LOGIC.md updates beside changed + compiler/runtime code, format-doc builtin table final, board + story + rows updated with measured numbers; commit locally. + +## Out of scope — deferred by name + +- Everything spec section 7 lists: set operators, outer joins, subqueries, + composite group keys, groups as values, FK cascade/set-nil, deferred + checks, SQL text in any role, cross-shard queries, `LIVE`, migrations, + cost-based planning. +- Sorted-grouping and partial-sort optimizations (recorded LINQ/postgres + precedents; hash + full stable sort are this plan's only strategies). +- The ecommerce sample's query rewrite (9b's fifth acceptance criterion) — + it lands as its own follow-up once the employee gate is green, so this + plan's blast radius stays one new sample. diff --git a/docs/runtime/database/02-wo-language.md b/docs/runtime/database/02-wo-language.md index 1b6ba25..548a217 100644 --- a/docs/runtime/database/02-wo-language.md +++ b/docs/runtime/database/02-wo-language.md @@ -2,6 +2,14 @@ > A two-layer multi-paradigm language for an e-commerce platform with ACID transactions across relational, document, and graph storage. +> **Query layer superseded (2026-08-15):** the SQL + Cypher query layer below +> is design history — on the C stack, programs query their tables through the +> language-integrated surface specified in +> [`2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md) +> (fork 1's decision record). This document remains the reference for the +> schema layer's vocabulary and for the `wo-db` prototype's engine semantics; +> SQL text is at most a future export format, never a program surface. + **Previous**: [Phase 1 — Database Evaluation](./01-evaluation.md) | **Next**: [Phase 3 — In-Memory Engine](./03-inmemory-engine.md) | **Index**: [database.md](../database.md) --- diff --git a/docs/stories/language-runtime-database/09b-table-relations-query.md b/docs/stories/language-runtime-database/09b-table-relations-query.md index 8c71914..ab8c6f1 100644 --- a/docs/stories/language-runtime-database/09b-table-relations-query.md +++ b/docs/stories/language-runtime-database/09b-table-relations-query.md @@ -8,9 +8,14 @@ > precedes iteration 10 because `service` blocks will want to return query > results. > -> **No spec exists yet.** This iteration frames the outcome and records the -> open questions; the design must be brainstormed before a plan is written. -> The three questions in *Info* are genuine forks, not details. +> **Spec exists (2026-08-15):** +> [`2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md) +> settles the three forks recorded in *Info* below (kept as the decision +> record): the SQL/Cypher layer is superseded as the program surface, +> the syntax is a compiler-desugared comprehension, and the references +> contribute vocabulary + semantics (System.Linq) and execution + integrity +> vocabulary (PostgreSQL, surveyed with the spec). Plan: +> [`2026-08-15-employee-relations-query.md`](../../plan/compiler/2026-08-15-employee-relations-query.md). ## Goals @@ -119,10 +124,18 @@ iteration makes it mean something in the C stack. ## Proposed Solution -- **Brainstorm a spec first**, settling the three forks above; only then write - the plan. This iteration deliberately ships no plan pointer, because - choosing between "replace the SQL layer" and "sit beside it" changes what - the plan contains. +- ~~Brainstorm a spec first~~ — **done 2026-08-15**; the spec settles all + three forks and the plan exists (pointers in the header note). The fork-1 + outcome for the record: language-integrated query is the only program + surface; `docs/runtime/database/02-wo-language.md`'s SQL/Cypher layer stays + as design history and as the `wo-db` prototype's engine-semantics + reference, never as syntax. +- **The acceptance workload is a new sample**: `docs/examples/employee` — + `Department`/`Employee` with `@unique`, a composite index, a `ref`/ + `backlink` pair, and a report mode that is one `GROUP BY` after another + (headcount, avg/min/max salary by department). It is 9b's acceptance the + way log-watcher was iterations 1–7's; the ecommerce query rewrite (the + fifth criterion below) follows as its own step once employee is green. - Study `.dev/reference/dotnet-runtime`'s `System.Linq` operator set for the vocabulary, and `docs/runtime/database/02-wo-language.md` plus `prototypes/wo-db/` for the semantics already committed to. diff --git a/docs/superpowers/specs/2026-08-15-table-relations-query-design.md b/docs/superpowers/specs/2026-08-15-table-relations-query-design.md new file mode 100644 index 0000000..a8db8cc --- /dev/null +++ b/docs/superpowers/specs/2026-08-15-table-relations-query-design.md @@ -0,0 +1,307 @@ +# `@table`, Relations, and Language-Integrated Query — Design + +> **Status: proposed** (story iteration 9b). Settles the three forks recorded in +> [`09b-table-relations-query.md`](../../stories/language-runtime-database/09b-table-relations-query.md). +> Plan: [`docs/plan/compiler/2026-08-15-employee-relations-query.md`](../../plan/compiler/2026-08-15-employee-relations-query.md). +> Depends on iteration 9's engine plan +> ([`2026-08-01-db-engine-binding.md`](../plans/2026-08-01-db-engine-binding.md)) +> for storage, WAL, insert and the select subset. + +**One sentence:** queries are written in the language, checked by the compiler, +and lowered to bytecode loops over a handful of engine cursor builtins — no SQL +text exists anywhere in a compiled program — and the acceptance workload is a +new `docs/examples/employee` sample whose report mode is one `GROUP BY` after +another. + +--- + +## 1. The three forks, settled + +### Fork 1 — the SQL/Cypher layer is superseded as the program surface + +`docs/runtime/database/02-wo-language.md` specifies a query layer of literal +SQL and Cypher with five fixed-glue rules. That layer is **not built on the C +stack**. The language-integrated surface below is the only way a `.wo` program +queries its tables. + +Reasons, in order of weight: + +- The iteration's own acceptance forbids the alternative: a query must lower to + engine operations "not a string handed to a parser at runtime — provable by + disassembly." A resident SQL parser and a compile-checked surface would be + two grammars for one meaning, and the second grammar would re-introduce at + runtime every error class the first one eliminated at compile time. +- The language's identity is compile-time checking (the WO-E diagnostic + catalogue). "A typo in a field name is a compile error" cannot be delivered + by a text layer. +- A binary that ships a SQL parser it uses only for its own programs pays + image size and attack surface for nothing. + +What survives: `02-wo-language.md` stays as design history and as the +specification the `prototypes/wo-db` C++ prototype implements; that prototype +remains the **engine-semantics** reference (what an index probe returns, what +unique violation means), not a syntax reference. SQL text remains a candidate +*export/interop* format for much later (external tools speaking to a writeonce +service), explicitly not part of this iteration. + +### Fork 2 — comprehension syntax, desugared at compile time + +writeonce has no function values, so LINQ's method-chains-taking-lambdas are +unwritable. The surface is a **query expression** — a comprehension the +compiler desugars — where every predicate and projection is ordinary +expression syntax with the range variable in scope: + +```wo +let seniors = from e in Employee + where e.salary > 90000 + order by e.salary desc + select e; + +let by_dept = from e in Employee + group e by e.dept into g + select { dept: g.key.name, headcount: count(g), avg_salary: avg(g.salary) }; +``` + +(Illustrative; the grammar is normative in prose, section 3.) + +Why comprehension and not chaining: a chained `.where(e.salary > 100)` has no +binding site for `e` — the comprehension's `from e in` clause is what +introduces the variable, which is exactly the property that makes every later +clause an ordinary typed expression the existing typechecker can check. C# had +to add query expressions *on top of* lambdas for the same readability reason; +we get to skip the lambda layer entirely. The predicate is compile-time +syntax, never a runtime closure — which is also what lets the whole query +lower to plain bytecode. + +### Fork 3 — what each reference contributes + +**System.Linq** (surveyed 2026-08-15) contributes the operator vocabulary and +its semantic edge cases, not machinery: + +- The lowering target for `group … select aggregate` is the shape of + `AggregateBy`/`CountBy` and the `GroupBy(key, resultSelector)` overload: + **group-and-reduce as one node, no intermediate group object materialized** + (`Grouping.cs:63`, `AggregateBy.cs`). The `IGrouping`-returning overloads + exist to hand groups around as values; we have no delegates to hand them to, + so groups surface only as the `into g` binding inside the query itself. +- Empty-source rules per aggregate (section 4's table) adapt LINQ's — where + C# throws `InvalidOperationException`, writeonce answers `nil` through a + `?T` result, because an empty group is data, not a fault. +- Ordering is **stable**, guaranteed — LINQ enforces it with an original-index + tiebreak (`OrderedEnumerable.cs:432`); we adopt the same guarantee and the + same escape hatch (an unstable sort is legal when the key is a scalar and + rows are not identity-bearing, `OrderBy.cs:144-163` precedent). +- Join semantics: build a hash on the inner side, probe with the outer, and + **nil never joins** — LINQ achieves SQL's NULL-never-matches by refusing to + insert null keys into the build side (`Lookup.cs:119-122`); we do the same. + In grouping, by contrast, nil **is** a legitimate key (`Lookup.cs:207`). + +**PostgreSQL** (surveyed 2026-08-15) contributes execution vocabulary for the +engine side: + +- Aggregate execution is the transition/finalize split from `nodeAgg.c`: a + per-group transition value advanced once per row, a finalize step converting + state to result (`avg` carries sum+count without the executor knowing). The + `noTransValue` vs `transValueIsNull` distinction — "no row seen yet" is not + "the value is nil" — is adopted verbatim; it is what makes nil-skipping + aggregates correct without special-casing the first row. +- Grouping strategy: hash aggregation (`AGG_HASHED`) is the only strategy this + iteration builds; sorted grouping is an optimization for later. Project the + input to only the columns the aggregate needs before hashing + (`find_hash_columns` precedent). +- Referential integrity semantics from `ri_triggers.c`, mechanism discarded: + an FK check is a **direct probe of the referenced table's primary index** + (never query text); a nil `ref` passes the check (MATCH SIMPLE rule); an + update that does not change the key skips the check + (`RI_FKey_fk_upd_check_required` precedent). Enforcement action is + **restrict only** — deleting a Department that still has Employees traps; + cascade/set-nil are out of scope. +- Everything MVCC, buffer-manager, lock-manager and cost-planner shaped is + explicitly non-transferable: single-writer-per-shard RAM-authoritative + storage designed those problems away (iteration 9's doctrine). + +--- + +## 2. Relations become typed navigation + +The declaration vocabulary already exists in the samples and partially in the +compiler; this iteration makes it mean something: + +- `dept: ref Department` — a foreign key. Stored as the target's row id (a + scalar column, already the compiler's classification). **Navigating** it in + an expression (`e.dept.name`) typechecks with `Department`'s field set in + scope and lowers to a primary-index point read. +- `staff: backlink Employee.dept` — the declared inverse. Not a stored column; + reading it (`d.staff`) is a secondary-index scan of `Employee`'s `dept` + column and yields `multi Employee`. Declaring a `backlink` whose target + field is not a `ref` to this class is a compile error. +- `?ref Department` — an optional relation; nil stores as id 0, never probes, + never joins. + +Integrity, enforced at the engine's row choke points (iteration 9's index +doctrine — nothing touches storage except the row API): + +- Insert/update of a non-nil `ref`: probe the referenced primary index; a miss + traps with a foreign-key violation code (sibling of iteration 9's + unique-violation trap). +- Delete of a row that a non-nil `ref` still points at: trap (restrict). The + check is a probe of the secondary index that the `backlink` already + requires, so restrict costs one lookup and no new structure. + +--- + +## 3. The query surface (normative, in prose) + +A **query expression** is an expression. Its clauses, in the only order they +may appear: + +1. `from in ` — required, first. Source is a table (class name), + a `backlink` navigation, or a `multi` value. Introduces the range variable. +2. `where ` — optional, repeatable. Ordinary boolean expression + over range variables; each `where` is a filter. +3. `join in on == ` — optional. Equi-join only; + one side references only the outer variable, the other only the joined one. + Most joins in practice are spelled as `ref` navigation instead; explicit + `join` exists for joining on non-relation columns. +4. `group by into ` — optional. Ends the scope of the + range variables and opens the scope of `g`. `g.key` is the key's value. + For any field `f` of the grouped element, `g.f` names the **column of + members** — legal only inside an aggregate call. Nil is a legitimate key. +5. `order by [desc]` — optional, comma-repeatable keys. Stable. +6. `take ` / `skip ` — optional. +7. `select ` — required, last. The result element: a whole row, a single + field, or a projection literal `{ name: expr, … }` whose type the compiler + synthesizes as an anonymous record. Using a column the projection dropped + is a compile error thereafter (the 9b acceptance's projection criterion). + +A query's value is `multi `; a query wrapped directly in a whole- +query aggregate (`count(from …)`) is that aggregate's scalar. Execution is +eager at the point the expression is evaluated — no deferred queries, no query +values passed around (that would be a function value in disguise). + +Aggregates are compiler-recognized **clause functions**, legal over a group +binding or a whole query: `count(g)`, `count(from …)`, `sum(g.f)`, `avg(g.f)`, +`min(g.f)`, `max(g.f)`. They are not general functions; naming one outside a +query is the existing unknown-identifier error. + +Diagnostics this surface owns (new WO-E5xx range): unknown table, unknown +column (naming the class), relation navigated through a non-`ref` field, +aggregate outside a query, group member column used bare outside an aggregate, +join sides not separable, projection field name collision. + +--- + +## 4. Aggregate semantics (normative table) + +| Aggregate | Input | Result type | Empty source | Nil elements (`?T` column) | +|---|---|---|---|---| +| `count(g)` | group/query | `Int` | `0` | counted (row exists) | +| `sum(g.f)` | `Int` column | `Int` | `0` | skipped | +| `avg(g.f)` | `Int` column | `?Int` | `nil` | skipped; all-nil ⇒ `nil` | +| `min(g.f)` / `max(g.f)` | `Int` or `Text` column | `?T` | `nil` | skipped; all-nil ⇒ `nil` | + +Decisions behind the table: + +- **Empty is data, not a fault.** LINQ throws on empty `Average`/`Min`/`Max` + over non-nullables; writeonce has `?T` and a nil-forcing style already, so + the nullable-column LINQ behavior (`null` on empty, skip nils, all-nil ⇒ + `null` — `Average.cs:165`, `Min.cs:47-56`) is the behavior for everyone. + Traps stay reserved for integrity violations. +- **`sum` wraps.** The VM's ADD wraps two's-complement; `sum` is a loop of + ADDs and inherits that. Documented, consistent with language arithmetic, + and the alternative (checked overflow, LINQ's `Sum.cs:40`) would make `sum` + the only trapping arithmetic in the language. +- **`avg` truncates** toward zero (integer division), same as `/`. +- Execution is the transition/finalize ABI (section 1, fork 3): one opaque + transition slot per (group, aggregate), advanced per row, finalized per + group when the hash table drains. + +--- + +## 5. Lowering and execution model + +A query compiles to an **ordinary bytecode loop** — the same opcodes every +`for` loop uses — over a small set of new engine cursor builtins. There is no +plan-tree interpreter in the VM: the compiler *is* the planner, at the only +scale this iteration promises (index selection, no cost model). + +- **Scan**: table cursor builtins (open by class id, advance, read row into a + register as a borrowed row view). `where` clauses are ordinary compiled + boolean expressions guarding the loop body — the flattened-expression + lesson of PostgreSQL's executor (`execExpr.c`) is in our case simply "the + bytecode we already generate." +- **Index selection**: if the leading `where` conjuncts equality-match a + declared index's prefix (longest-prefix rule, iteration 9's `find_by`), the + compiler emits an index-probe cursor instead of a scan, and the acceptance + demonstrates the difference (a probe counter the engine exposes in debug + builds — "demonstrated, not assumed"). +- **`ref` navigation** lowers to a primary-index point-read builtin. + **`backlink`** lowers to a secondary-index cursor over the ref column. +- **`group … into g … select`** lowers to the two-phase hash-aggregation + shape: phase one iterates the source, keying a group hash table and + advancing transition slots (builtins: group-table create / upsert-advance); + phase two drains the table, runs finalize, evaluates the `select` + projection per group. One node, no intermediate group objects — the + `AggregateBy` shape. +- **`join`** lowers to build-inner/probe-outer against a transient hash + keyed on the join column; nil keys are never inserted and never probe. +- **`order by`** materializes the result `multi` and stable-sorts it; `take`/ + `skip` slice afterwards (a partial-sort fast path is recorded as a later + optimization, LINQ `SpeedOpt` precedent). +- **Ownership**: rows read from a table are engine-owned; anything a query + *returns* is copied out at the select boundary under Task-2's established + rule — a Text crossing an ownership boundary is copied, a projection record + is freshly built and caller-owned. Queries introduce no new ownership + classes. + +Format consequences: new builtin ids appended to the format doc's table (the +cursor/group/probe set), no new opcodes, no version bump beyond iteration 9's. + +--- + +## 6. Acceptance workload — `docs/examples/employee` + +A new sample, structured like `log-watcher` (wo.toml manifest, program mode, +`just` module, acceptance script), small enough to read in one sitting and +shaped so every 9b feature is load-bearing: + +- **Types**: `Department` (`@table(name: "departments", index: [name])`, + `name: Text @unique`) and `Employee` (`@table(name: "employees", + index: [dept], index: [dept, salary])`, `name: Text`, `salary: Int` cents, + `hired: Int` ms, `dept: ref Department`); `Department.staff: backlink + Employee.dept`. +- **Modes** (CLI, exit codes per the program-mode contract): + - `seed` — inserts departments and employees; proves insert + WAL + unique + trap (second `seed` run reports the duplicate-department trap caught). + - `report` — the GroupBy showcase: headcount by department, average / + min / max salary by department, total payroll, departments ordered by + average salary — each line's expected output is byte-exact in the + acceptance script. + - `staff ` — relation navigation both directions: finds the + department by unique name (index probe), lists its `staff` backlink + ordered by salary; also prints each employee's `e.dept.name` to prove + forward navigation. + - `raise ` — update through a query result; re-running + `report` shows the moved averages; a `raise` for a missing department + exercises the empty-query path (`avg` ⇒ nil). +- **Integrity demo**: deleting a department with staff traps (restrict); + the acceptance asserts the trap code. +- **Crash step**: kill -9 between `seed` and `report`, re-run `report` — + iteration 9's WAL replay proven on this workload too. + +The sample is 9b's acceptance the way log-watcher was iterations 1–7's: no new +corpus fixtures beyond the db corpus iteration 9 already plans; the sample is +the test. + +## 7. Out of scope (inherited and new) + +- Everything 9b's story already excludes: cross-shard queries and distributed + joins, `LIVE` subscriptions, migrations, cost-based planning. +- The `IGrouping`-as-value surface (groups escaping their query), `Distinct`/ + set operators, outer joins (`LeftJoin` family), subqueries in `where`, and + `group by` composite keys beyond a single expression — each waits for a + workload that demands it. +- FK actions other than restrict (cascade, set-nil), deferred constraint + checking (the after-trigger queue pattern is recorded for when transactions + span statements). +- SQL text in any runtime role.