shoney/writeonce
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

docs: iteration 9b spec + plan — @table, relations, query (employee)

Browse source 
- spec settles 9b's three forks: SQL/Cypher layer superseded as the
  program surface (design history + wo-db engine-semantics reference);
  comprehension syntax desugared at compile time (no function values);
  System.Linq = operator vocabulary + edge cases, PostgreSQL = execution
  + integrity vocabulary (both references surveyed 2026-08-15)
- aggregate semantics normative: count/sum total 0 on empty, avg/min/max
  are ?T with nil (empty is data, not a fault); nil skipped; sum wraps
  like language arithmetic; GroupBy lowers as group-and-reduce
  (AggregateBy shape), transition/finalize ABI from nodeAgg
- relations: ref = FK with direct-index-probe check (nil passes,
  unchanged-key skips), backlink = secondary-index scan, delete is
  restrict-only; nil never joins, nil is a legal group key
- lowering: the compiler is the planner — queries become bytecode loops
  over cursor/group builtins, longest-prefix index selection, no plan
  tree, no SQL text in the image (disassembly-provable)
- plan: 6 tasks gated by a new docs/examples/employee sample
  (Department/Employee, @unique, composite index, ref/backlink,
  GroupBy report mode) with its own acceptance script + crash step;
  blocked on iteration 9's engine plan
- story 09b + status board updated; 02-wo-language.md carries the
  supersession note

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
shoney.arickathil 2026-08-15 02:00:01 +02:00
parent 5e27b2f685
commit 5943bf6890
5 changed files with 537 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
docs/00-status.md
View file

@ -116,7 +116,7 @@ that sequences its tasks. Read one, approve, then the next starts.
| 7b | [Inferred GC + mark-sweep](stories/language-runtime-database/07b-inferred-gc-mark-sweep.md) | ⏸ off the workload's path (no `@gc`) |
| 8 | [Shard-actor runtime](stories/language-runtime-database/08-shard-actor-runtime.md) | ⬜ |
| 9 | [Database engine](stories/language-runtime-database/09-database-engine.md) | ⬜ |
| 9b | [`@table`, relations, query](stories/language-runtime-database/09b-table-relations-query.md) | ⬜ needs a spec first |
| 9b | [`@table`, relations, query](stories/language-runtime-database/09b-table-relations-query.md) | ⬜ spec + plan ready (2026-08-15) |
| 10 | [HTTP service layer](stories/language-runtime-database/10-http-service.md) | ⬜ | Hold |
| 11 | [Fibers](stories/language-runtime-database/11-fibers.md) | ⬜ | Hold |
| 12 | [Blue-green deploy](stories/language-runtime-database/12-blue-green-deploy.md) | ⬜ | Hold |
@ -322,7 +322,7 @@ Ecommerce sample (verified 2026-06-13): `api.rest` 17/17 expected statuses pass.
| 7b | Inferred GC + incremental mark-sweep — `@gc` removed, GC-ness inferred, RC retired | [spec](superpowers/specs/2026-08-11-inferred-gc-mark-sweep-design.md) — plan to be written |
| 8 | Shard-actor runtime | [plan 4](superpowers/plans/2026-08-01-shard-actor-vm-runtime.md) |
| 9 | Database engine binding | [plan 5](superpowers/plans/2026-08-01-db-engine-binding.md) |
| 9b | `@table` + relations + language-integrated query | **no spec yet** — three open forks recorded in the iteration; brainstorm before planning |
| 9b | `@table` + relations + language-integrated query — comprehension queries, `ref`/`backlink` navigation, GroupBy aggregates; acceptance: new `docs/examples/employee` sample | [spec](superpowers/specs/2026-08-15-table-relations-query-design.md) · [plan](plan/compiler/2026-08-15-employee-relations-query.md) |
| 10 | HTTP service layer | [plan 6](superpowers/plans/2026-08-01-http-service-layer.md) |
| 11 | Fibers | vision §3, [blue-green exploration](plan/exploration/blue-green-vm/00-vision.md) |
| 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9–10 |

200
docs/plan/compiler/2026-08-15-employee-relations-query.md Normal file
View file

@ -0,0 +1,200 @@
# `@table`, Relations, Query — Implementation Plan (employee sample)
> **Status: ⬜ pending** (story iteration 9b) — blocked on iteration 9's engine
> plan ([`2026-08-01-db-engine-binding.md`](../../superpowers/plans/2026-08-01-db-engine-binding.md)):
> Tasks 3–6 below consume its row storage, WAL, indexes and select subset.
> Board: [00-status.md](../../00-status.md)
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
>
> **Style rule (user convention):** concept, reason, and required behavior in words only; the executor writes the code.
**Spec:** [`docs/superpowers/specs/2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md) (normative: the settled forks, the clause grammar, the aggregate semantics table, the lowering model), amended by the systems-track spec's program-mode contract for the sample's CLI.
**Goal:** `@table` classes queried **in the language**: comprehension queries
with `where`/`join`/`group`/`order`/`take`/`select`, typed `ref`/`backlink`
navigation with restrict integrity, and the LINQ aggregate vocabulary
(`count`/`sum`/`avg`/`min`/`max`, `GroupBy` as group-and-reduce) — all lowered
to bytecode loops over engine cursor builtins, proven by a new
`docs/examples/employee` sample whose acceptance script is the gate.
**Architecture:** compiler front (`compiler/src/{lexer,parser,ast,types,owner,emit}.ml`)
for the query surface; runtime (`runtime/src/{table,db,builtin}.c` from
iteration 9, plus a new `query.c` for cursors and group hashing). The compiler
is the planner: index selection happens at lowering, the VM never sees a plan
tree. Reference semantics: System.Linq for operator meaning, PostgreSQL's
nodeAgg/ri_triggers for execution and integrity vocabulary (both surveyed in
the spec, fork 3).
**Tech Stack:** OCaml stdlib (compiler), C11 libc (runtime). No new opcodes;
new builtin ids appended to the format doc.
## Global Constraints
- **The sample is the test.** `docs/examples/employee` plus its acceptance
script is 9b's gate; the only corpus additions are the db-corpus fixtures
iteration 9 already plans. `just oop-e2e`, `just woc-test`, `just wovm-test`
stay green as regression after every task.
- **Index doctrine** (iteration 9, verbatim): secondary indexes are maintained
only through the engine's row choke points; FK checks and backlink reads are
index probes, never storage walks.
- **No SQL text anywhere** — the 9b acceptance's disassembly criterion.
- **No function values.** Every predicate/projection is expression syntax; a
query value is never deferred or passed around.
- Plans/specs in words; commits local only, never push; docs under `docs/`;
CODE-LOGIC.md updated beside changed code.
## File Structure
```
compiler/src/lexer.ml parser.ml ast.ml query-expression grammar, clause AST (Task 1)
compiler/src/types.ml range/group scopes, navigation, projection synthesis (Task 2)
compiler/src/owner.ml emit.ml query ownership + lowering to cursor builtins (Task 5)
runtime/src/query.c query.h cursors, group hash, transition/finalize aggregates (Tasks 3, 4)
runtime/src/db.c table.c FK restrict checks at the row choke points (Task 3)
docs/examples/employee/ the acceptance workload (Task 6)
scripts/employee-accept.sh the gate (Task 6)
docs/plan/oop-vm/00-wob-format.md appended builtin ids (Tasks 3-5)
```
---
### Task 1: Query expressions parse
**Concept & reason:** the comprehension grammar from spec section 3 —
`from`/`where`/`join`/`group…into`/`order by`/`take`/`skip`/`select`, clause
order fixed, aggregates as clause functions — becomes lexer keywords (contextual,
so `from`/`group`/`order` stay legal identifiers outside a query), a clause AST,
and parse diagnostics in a new WO-E5xx range (clause out of order, missing
`select`, aggregate named outside a query). Fixed clause order is a parse rule,
not a type rule, so the error lands on the exact token.
- [ ] Grammar and AST for every clause; contextual keywords verified against
the existing samples (no `.wo` file in the tree breaks).
- [ ] WO-E5xx diagnostics with golden coverage in the existing `woc-test`
suite (dump-ast goldens for well-formed queries, diagnostic goldens for
each malformed shape).
- [ ] Gates green; commit locally.
### Task 2: Queries typecheck
**Concept & reason:** the surface's whole promise is compile-time checking.
`from e in Employee` opens a scope where `e`'s fields are the class's; `ref`
navigation substitutes the target class's field set (`e.dept.name`);
`backlink` reads type as `multi` of the source class; `group … into g` closes
the range scope and opens the group scope, where `g.key` has the key's type
and `g.f` is legal **only** inside an aggregate call; `select { … }`
synthesizes an anonymous record type so later use of a dropped column is a
compile error. Aggregate result types follow the spec's table exactly —
`count` is Int, `sum` Int, `avg`/`min`/`max` are `?T` because an empty group
is data. Unknown table, unknown column (naming the class), navigation through
a non-`ref`, bare group-member column, join sides not separable: each is its
own WO-E5xx with a golden.
- [ ] Scope machinery for range/group variables; navigation typing both
directions; projection record synthesis interned like other class
shapes.
- [ ] Aggregate typing per the spec table; `?T` results force the existing
nil-handling style at use sites.
- [ ] Diagnostic goldens for every error class named above; gates green;
commit locally.
### Task 3: Cursors and integrity in the engine
**Concept & reason:** the runtime side queries need, built on iteration 9's
storage: a table-scan cursor (open by class id, advance, borrowed row view),
a primary-index point read (`ref` navigation), a secondary-index cursor with
longest-prefix probe (backlink reads, indexed `where`), all as builtins
appended to the format doc. Integrity lands at the row choke points where the
indexes already live: inserting/updating a non-nil `ref` probes the referenced
primary index and traps on a miss (foreign-key violation, sibling of the
unique trap); deleting a row still referenced traps (restrict) via the same
secondary index a `backlink` requires — one probe, no new structure. Nil
`ref` never probes (the MATCH SIMPLE rule); an update that leaves the key
unchanged skips the check (both from the PostgreSQL RI survey, mechanism
discarded — a check is an index probe, never query text).
- [ ] Cursor builtins + borrowed-row-view lifetime rules written into the
binding doc (a row view never escapes the loop that opened the cursor —
the ownership pass enforces it, mirror of the container-read borrow).
- [ ] FK trap + restrict trap wired through the row API; a debug-build probe
counter exposed for Task 5's index-selection proof.
- [ ] db-corpus fixtures from iteration 9's plan extended with one FK-violation
and one restrict fixture (trap-code exact); ASan green; commit locally.
### Task 4: Group hash + aggregate execution
**Concept & reason:** the `AggregateBy` shape from the spec — group-and-reduce
in one pass, no group objects. A group hash table builtin set: create (keyed
by the group key's kind, nil legal), upsert-advance (locate-or-create the
group, advance each aggregate's transition slot), drain (iterate groups,
finalize, hand key + finals to the compiled projection loop). Transition/
finalize is PostgreSQL's split: `avg` carries sum+count in its transition
state and divides at finalize; the "no row seen yet" state is distinct from
"transition value is nil" so nil-skipping aggregates need no first-row special
case. Aggregate semantics are the spec's table: empty ⇒ nil (or 0 for
count/sum), nil elements skipped, sum wraps, avg truncates.
- [ ] Builtins implemented; the input projected to only the columns the
aggregates read before hashing (the nodeAgg memory lesson).
- [ ] Fixture-level verification through iteration 9's db corpus (one grouped
query, exact output) plus the sample in Task 6; ASan green; commit
locally.
### Task 5: Lowering — the compiler is the planner
**Concept & reason:** a query desugars to the bytecode loops the language
already has. Scan or probe chosen at compile time: leading `where` equality
conjuncts matched against declared indexes longest-prefix-first, probe emitted
on a hit, scan otherwise — and the choice is **demonstrated** via Task 3's
probe counter in the acceptance, not assumed. `join` builds a transient hash
on the inner side and probes with the outer (nil never inserted, never
probes). `group` emits the two-phase hash-aggregation loop from Task 4.
`order by` materializes and stable-sorts (original-index tiebreak — the LINQ
stability guarantee); `take`/`skip` slice the result. Ownership: row views
stay loop-bound borrows; everything `select` emits is copied/built at the
boundary under the established Text-copy and fresh-value rules — queries add
no new ownership classes, and the ownership pass's existing drop machinery
covers the query's temporaries because the lowering IS ordinary loops.
- [ ] Desugar + lowering for every clause; disassembly of the sample's report
mode shows loops and builtins, no plan tree, no text.
- [ ] Index selection proven: the acceptance asserts probe-counter deltas for
the indexed `staff <department>` path versus a full-scan query.
- [ ] `oop-e2e`, `woc-test`, ASan-corpus gates green; commit locally.
### Task 6: The employee sample is the acceptance
**Concept & reason:** spec section 6 verbatim — `docs/examples/employee` with
`Department`/`Employee` (`@table`, `@unique` name, composite `[dept, salary]`
index, `ref`/`backlink` pair), wo.toml manifest so `woc .` builds it, a `just`
module beside it (the log-watcher convention), and `scripts/employee-accept.sh`
as the gate: `seed` (insert + WAL + duplicate-department trap on the second
run), `report` (headcount/avg/min/max by department, total payroll, ordered by
average salary — byte-exact lines), `staff` (both navigation directions +
index-probe proof), `raise` (update through a query; missing department ⇒ the
empty-query nil path), the restrict-trap demo, and a kill -9 between `seed`
and `report` proving replay on this workload.
- [ ] Sample written; `woc docs/examples/employee` compiles with zero
diagnostics; the binary's modes run.
- [ ] `scripts/employee-accept.sh` + `just employee` module: every mode
checked with exact expectations, trap codes asserted, crash step
included; soak-style RSS/fd sampling reused from the log-watcher
script's pattern for the `report` loop.
- [ ] ASan run of the full acceptance: zero leaks (the log-watcher bar).
- [ ] Docs: README beside the sample, CODE-LOGIC.md updates beside changed
compiler/runtime code, format-doc builtin table final, board + story
rows updated with measured numbers; commit locally.
## Out of scope — deferred by name
- Everything spec section 7 lists: set operators, outer joins, subqueries,
composite group keys, groups as values, FK cascade/set-nil, deferred
checks, SQL text in any role, cross-shard queries, `LIVE`, migrations,
cost-based planning.
- Sorted-grouping and partial-sort optimizations (recorded LINQ/postgres
precedents; hash + full stable sort are this plan's only strategies).
- The ecommerce sample's query rewrite (9b's fifth acceptance criterion) —
it lands as its own follow-up once the employee gate is green, so this
plan's blast radius stays one new sample.

8
docs/runtime/database/02-wo-language.md
View file

@ -2,6 +2,14 @@
> A two-layer multi-paradigm language for an e-commerce platform with ACID transactions across relational, document, and graph storage.
> **Query layer superseded (2026-08-15):** the SQL + Cypher query layer below
> is design history — on the C stack, programs query their tables through the
> language-integrated surface specified in
> [`2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md)
> (fork 1's decision record). This document remains the reference for the
> schema layer's vocabulary and for the `wo-db` prototype's engine semantics;
> SQL text is at most a future export format, never a program surface.
**Previous**: [Phase 1 — Database Evaluation](./01-evaluation.md) | **Next**: [Phase 3 — In-Memory Engine](./03-inmemory-engine.md) | **Index**: [database.md](../database.md)
---

27
docs/stories/language-runtime-database/09b-table-relations-query.md
View file

@ -8,9 +8,14 @@
> precedes iteration 10 because `service` blocks will want to return query
> results.
>
> **No spec exists yet.** This iteration frames the outcome and records the
> open questions; the design must be brainstormed before a plan is written.
> The three questions in *Info* are genuine forks, not details.
> **Spec exists (2026-08-15):**
> [`2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md)
> settles the three forks recorded in *Info* below (kept as the decision
> record): the SQL/Cypher layer is superseded as the program surface,
> the syntax is a compiler-desugared comprehension, and the references
> contribute vocabulary + semantics (System.Linq) and execution + integrity
> vocabulary (PostgreSQL, surveyed with the spec). Plan:
> [`2026-08-15-employee-relations-query.md`](../../plan/compiler/2026-08-15-employee-relations-query.md).
## Goals
@ -119,10 +124,18 @@ iteration makes it mean something in the C stack.
## Proposed Solution
- **Brainstorm a spec first**, settling the three forks above; only then write
the plan. This iteration deliberately ships no plan pointer, because
choosing between "replace the SQL layer" and "sit beside it" changes what
the plan contains.
- ~~Brainstorm a spec first~~ — **done 2026-08-15**; the spec settles all
three forks and the plan exists (pointers in the header note). The fork-1
outcome for the record: language-integrated query is the only program
surface; `docs/runtime/database/02-wo-language.md`'s SQL/Cypher layer stays
as design history and as the `wo-db` prototype's engine-semantics
reference, never as syntax.
- **The acceptance workload is a new sample**: `docs/examples/employee` —
`Department`/`Employee` with `@unique`, a composite index, a `ref`/
`backlink` pair, and a report mode that is one `GROUP BY` after another
(headcount, avg/min/max salary by department). It is 9b's acceptance the
way log-watcher was iterations 1–7's; the ecommerce query rewrite (the
fifth criterion below) follows as its own step once employee is green.
- Study `.dev/reference/dotnet-runtime`'s `System.Linq` operator set for the
vocabulary, and `docs/runtime/database/02-wo-language.md` plus
`prototypes/wo-db/` for the semantics already committed to.

307
docs/superpowers/specs/2026-08-15-table-relations-query-design.md Normal file
View file

@ -0,0 +1,307 @@
# `@table`, Relations, and Language-Integrated Query — Design
> **Status: proposed** (story iteration 9b). Settles the three forks recorded in
> [`09b-table-relations-query.md`](../../stories/language-runtime-database/09b-table-relations-query.md).
> Plan: [`docs/plan/compiler/2026-08-15-employee-relations-query.md`](../../plan/compiler/2026-08-15-employee-relations-query.md).
> Depends on iteration 9's engine plan
> ([`2026-08-01-db-engine-binding.md`](../plans/2026-08-01-db-engine-binding.md))
> for storage, WAL, insert and the select subset.
**One sentence:** queries are written in the language, checked by the compiler,
and lowered to bytecode loops over a handful of engine cursor builtins — no SQL
text exists anywhere in a compiled program — and the acceptance workload is a
new `docs/examples/employee` sample whose report mode is one `GROUP BY` after
another.
---
## 1. The three forks, settled
### Fork 1 — the SQL/Cypher layer is superseded as the program surface
`docs/runtime/database/02-wo-language.md` specifies a query layer of literal
SQL and Cypher with five fixed-glue rules. That layer is **not built on the C
stack**. The language-integrated surface below is the only way a `.wo` program
queries its tables.
Reasons, in order of weight:
- The iteration's own acceptance forbids the alternative: a query must lower to
engine operations "not a string handed to a parser at runtime — provable by
disassembly." A resident SQL parser and a compile-checked surface would be
two grammars for one meaning, and the second grammar would re-introduce at
runtime every error class the first one eliminated at compile time.
- The language's identity is compile-time checking (the WO-E diagnostic
catalogue). "A typo in a field name is a compile error" cannot be delivered
by a text layer.
- A binary that ships a SQL parser it uses only for its own programs pays
image size and attack surface for nothing.
What survives: `02-wo-language.md` stays as design history and as the
specification the `prototypes/wo-db` C++ prototype implements; that prototype
remains the **engine-semantics** reference (what an index probe returns, what
unique violation means), not a syntax reference. SQL text remains a candidate
*export/interop* format for much later (external tools speaking to a writeonce
service), explicitly not part of this iteration.
### Fork 2 — comprehension syntax, desugared at compile time
writeonce has no function values, so LINQ's method-chains-taking-lambdas are
unwritable. The surface is a **query expression** — a comprehension the
compiler desugars — where every predicate and projection is ordinary
expression syntax with the range variable in scope:
```wo
let seniors = from e in Employee
where e.salary > 90000
order by e.salary desc
select e;
let by_dept = from e in Employee
group e by e.dept into g
select { dept: g.key.name, headcount: count(g), avg_salary: avg(g.salary) };
```
(Illustrative; the grammar is normative in prose, section 3.)
Why comprehension and not chaining: a chained `.where(e.salary > 100)` has no
binding site for `e` — the comprehension's `from e in` clause is what
introduces the variable, which is exactly the property that makes every later
clause an ordinary typed expression the existing typechecker can check. C# had
to add query expressions *on top of* lambdas for the same readability reason;
we get to skip the lambda layer entirely. The predicate is compile-time
syntax, never a runtime closure — which is also what lets the whole query
lower to plain bytecode.
### Fork 3 — what each reference contributes
**System.Linq** (surveyed 2026-08-15) contributes the operator vocabulary and
its semantic edge cases, not machinery:
- The lowering target for `group … select aggregate` is the shape of
`AggregateBy`/`CountBy` and the `GroupBy(key, resultSelector)` overload:
**group-and-reduce as one node, no intermediate group object materialized**
(`Grouping.cs:63`, `AggregateBy.cs`). The `IGrouping`-returning overloads
exist to hand groups around as values; we have no delegates to hand them to,
so groups surface only as the `into g` binding inside the query itself.
- Empty-source rules per aggregate (section 4's table) adapt LINQ's — where
C# throws `InvalidOperationException`, writeonce answers `nil` through a
`?T` result, because an empty group is data, not a fault.
- Ordering is **stable**, guaranteed — LINQ enforces it with an original-index
tiebreak (`OrderedEnumerable.cs:432`); we adopt the same guarantee and the
same escape hatch (an unstable sort is legal when the key is a scalar and
rows are not identity-bearing, `OrderBy.cs:144-163` precedent).
- Join semantics: build a hash on the inner side, probe with the outer, and
**nil never joins** — LINQ achieves SQL's NULL-never-matches by refusing to
insert null keys into the build side (`Lookup.cs:119-122`); we do the same.
In grouping, by contrast, nil **is** a legitimate key (`Lookup.cs:207`).
**PostgreSQL** (surveyed 2026-08-15) contributes execution vocabulary for the
engine side:
- Aggregate execution is the transition/finalize split from `nodeAgg.c`: a
per-group transition value advanced once per row, a finalize step converting
state to result (`avg` carries sum+count without the executor knowing). The
`noTransValue` vs `transValueIsNull` distinction — "no row seen yet" is not
"the value is nil" — is adopted verbatim; it is what makes nil-skipping
aggregates correct without special-casing the first row.
- Grouping strategy: hash aggregation (`AGG_HASHED`) is the only strategy this
iteration builds; sorted grouping is an optimization for later. Project the
input to only the columns the aggregate needs before hashing
(`find_hash_columns` precedent).
- Referential integrity semantics from `ri_triggers.c`, mechanism discarded:
an FK check is a **direct probe of the referenced table's primary index**
(never query text); a nil `ref` passes the check (MATCH SIMPLE rule); an
update that does not change the key skips the check
(`RI_FKey_fk_upd_check_required` precedent). Enforcement action is
**restrict only** — deleting a Department that still has Employees traps;
cascade/set-nil are out of scope.
- Everything MVCC, buffer-manager, lock-manager and cost-planner shaped is
explicitly non-transferable: single-writer-per-shard RAM-authoritative
storage designed those problems away (iteration 9's doctrine).
---
## 2. Relations become typed navigation
The declaration vocabulary already exists in the samples and partially in the
compiler; this iteration makes it mean something:
- `dept: ref Department` — a foreign key. Stored as the target's row id (a
scalar column, already the compiler's classification). **Navigating** it in
an expression (`e.dept.name`) typechecks with `Department`'s field set in
scope and lowers to a primary-index point read.
- `staff: backlink Employee.dept` — the declared inverse. Not a stored column;
reading it (`d.staff`) is a secondary-index scan of `Employee`'s `dept`
column and yields `multi Employee`. Declaring a `backlink` whose target
field is not a `ref` to this class is a compile error.
- `?ref Department` — an optional relation; nil stores as id 0, never probes,
never joins.
Integrity, enforced at the engine's row choke points (iteration 9's index
doctrine — nothing touches storage except the row API):
- Insert/update of a non-nil `ref`: probe the referenced primary index; a miss
traps with a foreign-key violation code (sibling of iteration 9's
unique-violation trap).
- Delete of a row that a non-nil `ref` still points at: trap (restrict). The
check is a probe of the secondary index that the `backlink` already
requires, so restrict costs one lookup and no new structure.
---
## 3. The query surface (normative, in prose)
A **query expression** is an expression. Its clauses, in the only order they
may appear:
1. `from <var> in <source>` — required, first. Source is a table (class name),
a `backlink` navigation, or a `multi` value. Introduces the range variable.
2. `where <bool-expr>` — optional, repeatable. Ordinary boolean expression
over range variables; each `where` is a filter.
3. `join <var2> in <source2> on <expr> == <expr2>` — optional. Equi-join only;
one side references only the outer variable, the other only the joined one.
Most joins in practice are spelled as `ref` navigation instead; explicit
`join` exists for joining on non-relation columns.
4. `group <expr> by <key-expr> into <g>` — optional. Ends the scope of the
range variables and opens the scope of `g`. `g.key` is the key's value.
For any field `f` of the grouped element, `g.f` names the **column of
members** — legal only inside an aggregate call. Nil is a legitimate key.
5. `order by <expr> [desc]` — optional, comma-repeatable keys. Stable.
6. `take <int-expr>` / `skip <int-expr>` — optional.
7. `select <expr>` — required, last. The result element: a whole row, a single
field, or a projection literal `{ name: expr, … }` whose type the compiler
synthesizes as an anonymous record. Using a column the projection dropped
is a compile error thereafter (the 9b acceptance's projection criterion).
A query's value is `multi <element>`; a query wrapped directly in a whole-
query aggregate (`count(from …)`) is that aggregate's scalar. Execution is
eager at the point the expression is evaluated — no deferred queries, no query
values passed around (that would be a function value in disguise).
Aggregates are compiler-recognized **clause functions**, legal over a group
binding or a whole query: `count(g)`, `count(from …)`, `sum(g.f)`, `avg(g.f)`,
`min(g.f)`, `max(g.f)`. They are not general functions; naming one outside a
query is the existing unknown-identifier error.
Diagnostics this surface owns (new WO-E5xx range): unknown table, unknown
column (naming the class), relation navigated through a non-`ref` field,
aggregate outside a query, group member column used bare outside an aggregate,
join sides not separable, projection field name collision.
---
## 4. Aggregate semantics (normative table)
| Aggregate | Input | Result type | Empty source | Nil elements (`?T` column) |
|---|---|---|---|---|
| `count(g)` | group/query | `Int` | `0` | counted (row exists) |
| `sum(g.f)` | `Int` column | `Int` | `0` | skipped |
| `avg(g.f)` | `Int` column | `?Int` | `nil` | skipped; all-nil ⇒ `nil` |
| `min(g.f)` / `max(g.f)` | `Int` or `Text` column | `?T` | `nil` | skipped; all-nil ⇒ `nil` |
Decisions behind the table:
- **Empty is data, not a fault.** LINQ throws on empty `Average`/`Min`/`Max`
over non-nullables; writeonce has `?T` and a nil-forcing style already, so
the nullable-column LINQ behavior (`null` on empty, skip nils, all-nil ⇒
`null` — `Average.cs:165`, `Min.cs:47-56`) is the behavior for everyone.
Traps stay reserved for integrity violations.
- **`sum` wraps.** The VM's ADD wraps two's-complement; `sum` is a loop of
ADDs and inherits that. Documented, consistent with language arithmetic,
and the alternative (checked overflow, LINQ's `Sum.cs:40`) would make `sum`
the only trapping arithmetic in the language.
- **`avg` truncates** toward zero (integer division), same as `/`.
- Execution is the transition/finalize ABI (section 1, fork 3): one opaque
transition slot per (group, aggregate), advanced per row, finalized per
group when the hash table drains.
---
## 5. Lowering and execution model
A query compiles to an **ordinary bytecode loop** — the same opcodes every
`for` loop uses — over a small set of new engine cursor builtins. There is no
plan-tree interpreter in the VM: the compiler *is* the planner, at the only
scale this iteration promises (index selection, no cost model).
- **Scan**: table cursor builtins (open by class id, advance, read row into a
register as a borrowed row view). `where` clauses are ordinary compiled
boolean expressions guarding the loop body — the flattened-expression
lesson of PostgreSQL's executor (`execExpr.c`) is in our case simply "the
bytecode we already generate."
- **Index selection**: if the leading `where` conjuncts equality-match a
declared index's prefix (longest-prefix rule, iteration 9's `find_by`), the
compiler emits an index-probe cursor instead of a scan, and the acceptance
demonstrates the difference (a probe counter the engine exposes in debug
builds — "demonstrated, not assumed").
- **`ref` navigation** lowers to a primary-index point-read builtin.
**`backlink`** lowers to a secondary-index cursor over the ref column.
- **`group … into g … select`** lowers to the two-phase hash-aggregation
shape: phase one iterates the source, keying a group hash table and
advancing transition slots (builtins: group-table create / upsert-advance);
phase two drains the table, runs finalize, evaluates the `select`
projection per group. One node, no intermediate group objects — the
`AggregateBy` shape.
- **`join`** lowers to build-inner/probe-outer against a transient hash
keyed on the join column; nil keys are never inserted and never probe.
- **`order by`** materializes the result `multi` and stable-sorts it; `take`/
`skip` slice afterwards (a partial-sort fast path is recorded as a later
optimization, LINQ `SpeedOpt` precedent).
- **Ownership**: rows read from a table are engine-owned; anything a query
*returns* is copied out at the select boundary under Task-2's established
rule — a Text crossing an ownership boundary is copied, a projection record
is freshly built and caller-owned. Queries introduce no new ownership
classes.
Format consequences: new builtin ids appended to the format doc's table (the
cursor/group/probe set), no new opcodes, no version bump beyond iteration 9's.
---
## 6. Acceptance workload — `docs/examples/employee`
A new sample, structured like `log-watcher` (wo.toml manifest, program mode,
`just` module, acceptance script), small enough to read in one sitting and
shaped so every 9b feature is load-bearing:
- **Types**: `Department` (`@table(name: "departments", index: [name])`,
`name: Text @unique`) and `Employee` (`@table(name: "employees",
index: [dept], index: [dept, salary])`, `name: Text`, `salary: Int` cents,
`hired: Int` ms, `dept: ref Department`); `Department.staff: backlink
Employee.dept`.
- **Modes** (CLI, exit codes per the program-mode contract):
- `seed` — inserts departments and employees; proves insert + WAL + unique
trap (second `seed` run reports the duplicate-department trap caught).
- `report` — the GroupBy showcase: headcount by department, average /
min / max salary by department, total payroll, departments ordered by
average salary — each line's expected output is byte-exact in the
acceptance script.
- `staff <department>` — relation navigation both directions: finds the
department by unique name (index probe), lists its `staff` backlink
ordered by salary; also prints each employee's `e.dept.name` to prove
forward navigation.
- `raise <department> <pct>` — update through a query result; re-running
`report` shows the moved averages; a `raise` for a missing department
exercises the empty-query path (`avg` ⇒ nil).
- **Integrity demo**: deleting a department with staff traps (restrict);
the acceptance asserts the trap code.
- **Crash step**: kill -9 between `seed` and `report`, re-run `report` —
iteration 9's WAL replay proven on this workload too.
The sample is 9b's acceptance the way log-watcher was iterations 1–7's: no new
corpus fixtures beyond the db corpus iteration 9 already plans; the sample is
the test.
## 7. Out of scope (inherited and new)
- Everything 9b's story already excludes: cross-shard queries and distributed
joins, `LIVE` subscriptions, migrations, cost-based planning.
- The `IGrouping`-as-value surface (groups escaping their query), `Distinct`/
set operators, outer joins (`LeftJoin` family), subqueries in `where`, and
`group by` composite keys beyond a single expression — each waits for a
workload that demands it.
- FK actions other than restrict (cascade, set-nil), deferred constraint
checking (the after-trigger queue pattern is recorded for when transactions
span statements).
- SQL text in any runtime role.