docs: iteration 9b spec + plan — @table, relations, query (employee)
- spec settles 9b's three forks: SQL/Cypher layer superseded as the program surface (design history + wo-db engine-semantics reference); comprehension syntax desugared at compile time (no function values); System.Linq = operator vocabulary + edge cases, PostgreSQL = execution + integrity vocabulary (both references surveyed 2026-08-15) - aggregate semantics normative: count/sum total 0 on empty, avg/min/max are ?T with nil (empty is data, not a fault); nil skipped; sum wraps like language arithmetic; GroupBy lowers as group-and-reduce (AggregateBy shape), transition/finalize ABI from nodeAgg - relations: ref = FK with direct-index-probe check (nil passes, unchanged-key skips), backlink = secondary-index scan, delete is restrict-only; nil never joins, nil is a legal group key - lowering: the compiler is the planner — queries become bytecode loops over cursor/group builtins, longest-prefix index selection, no plan tree, no SQL text in the image (disassembly-provable) - plan: 6 tasks gated by a new docs/examples/employee sample (Department/Employee, @unique, composite index, ref/backlink, GroupBy report mode) with its own acceptance script + crash step; blocked on iteration 9's engine plan - story 09b + status board updated; 02-wo-language.md carries the supersession note Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
48429c5a4a
commit
7f9332f5d2
5 changed files with 537 additions and 9 deletions
|
|
@ -116,7 +116,7 @@ that sequences its tasks. Read one, approve, then the next starts.
|
||||||
| 7b | [Inferred GC + mark-sweep](stories/language-runtime-database/07b-inferred-gc-mark-sweep.md) | ⏸ off the workload's path (no `@gc`) |
|
| 7b | [Inferred GC + mark-sweep](stories/language-runtime-database/07b-inferred-gc-mark-sweep.md) | ⏸ off the workload's path (no `@gc`) |
|
||||||
| 8 | [Shard-actor runtime](stories/language-runtime-database/08-shard-actor-runtime.md) | ⬜ |
|
| 8 | [Shard-actor runtime](stories/language-runtime-database/08-shard-actor-runtime.md) | ⬜ |
|
||||||
| 9 | [Database engine](stories/language-runtime-database/09-database-engine.md) | ⬜ |
|
| 9 | [Database engine](stories/language-runtime-database/09-database-engine.md) | ⬜ |
|
||||||
| 9b | [`@table`, relations, query](stories/language-runtime-database/09b-table-relations-query.md) | ⬜ needs a spec first |
|
| 9b | [`@table`, relations, query](stories/language-runtime-database/09b-table-relations-query.md) | ⬜ spec + plan ready (2026-08-15) |
|
||||||
| 10 | [HTTP service layer](stories/language-runtime-database/10-http-service.md) | ⬜ | Hold |
|
| 10 | [HTTP service layer](stories/language-runtime-database/10-http-service.md) | ⬜ | Hold |
|
||||||
| 11 | [Fibers](stories/language-runtime-database/11-fibers.md) | ⬜ | Hold |
|
| 11 | [Fibers](stories/language-runtime-database/11-fibers.md) | ⬜ | Hold |
|
||||||
| 12 | [Blue-green deploy](stories/language-runtime-database/12-blue-green-deploy.md) | ⬜ | Hold |
|
| 12 | [Blue-green deploy](stories/language-runtime-database/12-blue-green-deploy.md) | ⬜ | Hold |
|
||||||
|
|
@ -322,7 +322,7 @@ Ecommerce sample (verified 2026-06-13): `api.rest` 17/17 expected statuses pass.
|
||||||
| 7b | Inferred GC + incremental mark-sweep — `@gc` removed, GC-ness inferred, RC retired | [spec](superpowers/specs/2026-08-11-inferred-gc-mark-sweep-design.md) — plan to be written |
|
| 7b | Inferred GC + incremental mark-sweep — `@gc` removed, GC-ness inferred, RC retired | [spec](superpowers/specs/2026-08-11-inferred-gc-mark-sweep-design.md) — plan to be written |
|
||||||
| 8 | Shard-actor runtime | [plan 4](superpowers/plans/2026-08-01-shard-actor-vm-runtime.md) |
|
| 8 | Shard-actor runtime | [plan 4](superpowers/plans/2026-08-01-shard-actor-vm-runtime.md) |
|
||||||
| 9 | Database engine binding | [plan 5](superpowers/plans/2026-08-01-db-engine-binding.md) |
|
| 9 | Database engine binding | [plan 5](superpowers/plans/2026-08-01-db-engine-binding.md) |
|
||||||
| 9b | `@table` + relations + language-integrated query | **no spec yet** — three open forks recorded in the iteration; brainstorm before planning |
|
| 9b | `@table` + relations + language-integrated query — comprehension queries, `ref`/`backlink` navigation, GroupBy aggregates; acceptance: new `docs/examples/employee` sample | [spec](superpowers/specs/2026-08-15-table-relations-query-design.md) · [plan](plan/compiler/2026-08-15-employee-relations-query.md) |
|
||||||
| 10 | HTTP service layer | [plan 6](superpowers/plans/2026-08-01-http-service-layer.md) |
|
| 10 | HTTP service layer | [plan 6](superpowers/plans/2026-08-01-http-service-layer.md) |
|
||||||
| 11 | Fibers | vision §3, [blue-green exploration](plan/exploration/blue-green-vm/00-vision.md) |
|
| 11 | Fibers | vision §3, [blue-green exploration](plan/exploration/blue-green-vm/00-vision.md) |
|
||||||
| 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9–10 |
|
| 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9–10 |
|
||||||
|
|
|
||||||
200
docs/plan/compiler/2026-08-15-employee-relations-query.md
Normal file
200
docs/plan/compiler/2026-08-15-employee-relations-query.md
Normal file
|
|
@ -0,0 +1,200 @@
|
||||||
|
# `@table`, Relations, Query — Implementation Plan (employee sample)
|
||||||
|
|
||||||
|
> **Status: ⬜ pending** (story iteration 9b) — blocked on iteration 9's engine
|
||||||
|
> plan ([`2026-08-01-db-engine-binding.md`](../../superpowers/plans/2026-08-01-db-engine-binding.md)):
|
||||||
|
> Tasks 3–6 below consume its row storage, WAL, indexes and select subset.
|
||||||
|
> Board: [00-status.md](../../00-status.md)
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
>
|
||||||
|
> **Style rule (user convention):** concept, reason, and required behavior in words only; the executor writes the code.
|
||||||
|
|
||||||
|
**Spec:** [`docs/superpowers/specs/2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md) (normative: the settled forks, the clause grammar, the aggregate semantics table, the lowering model), amended by the systems-track spec's program-mode contract for the sample's CLI.
|
||||||
|
|
||||||
|
**Goal:** `@table` classes queried **in the language**: comprehension queries
|
||||||
|
with `where`/`join`/`group`/`order`/`take`/`select`, typed `ref`/`backlink`
|
||||||
|
navigation with restrict integrity, and the LINQ aggregate vocabulary
|
||||||
|
(`count`/`sum`/`avg`/`min`/`max`, `GroupBy` as group-and-reduce) — all lowered
|
||||||
|
to bytecode loops over engine cursor builtins, proven by a new
|
||||||
|
`docs/examples/employee` sample whose acceptance script is the gate.
|
||||||
|
|
||||||
|
**Architecture:** compiler front (`compiler/src/{lexer,parser,ast,types,owner,emit}.ml`)
|
||||||
|
for the query surface; runtime (`runtime/src/{table,db,builtin}.c` from
|
||||||
|
iteration 9, plus a new `query.c` for cursors and group hashing). The compiler
|
||||||
|
is the planner: index selection happens at lowering, the VM never sees a plan
|
||||||
|
tree. Reference semantics: System.Linq for operator meaning, PostgreSQL's
|
||||||
|
nodeAgg/ri_triggers for execution and integrity vocabulary (both surveyed in
|
||||||
|
the spec, fork 3).
|
||||||
|
|
||||||
|
**Tech Stack:** OCaml stdlib (compiler), C11 libc (runtime). No new opcodes;
|
||||||
|
new builtin ids appended to the format doc.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- **The sample is the test.** `docs/examples/employee` plus its acceptance
|
||||||
|
script is 9b's gate; the only corpus additions are the db-corpus fixtures
|
||||||
|
iteration 9 already plans. `just oop-e2e`, `just woc-test`, `just wovm-test`
|
||||||
|
stay green as regression after every task.
|
||||||
|
- **Index doctrine** (iteration 9, verbatim): secondary indexes are maintained
|
||||||
|
only through the engine's row choke points; FK checks and backlink reads are
|
||||||
|
index probes, never storage walks.
|
||||||
|
- **No SQL text anywhere** — the 9b acceptance's disassembly criterion.
|
||||||
|
- **No function values.** Every predicate/projection is expression syntax; a
|
||||||
|
query value is never deferred or passed around.
|
||||||
|
- Plans/specs in words; commits local only, never push; docs under `docs/`;
|
||||||
|
CODE-LOGIC.md updated beside changed code.
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
compiler/src/lexer.ml parser.ml ast.ml query-expression grammar, clause AST (Task 1)
|
||||||
|
compiler/src/types.ml range/group scopes, navigation, projection synthesis (Task 2)
|
||||||
|
compiler/src/owner.ml emit.ml query ownership + lowering to cursor builtins (Task 5)
|
||||||
|
runtime/src/query.c query.h cursors, group hash, transition/finalize aggregates (Tasks 3, 4)
|
||||||
|
runtime/src/db.c table.c FK restrict checks at the row choke points (Task 3)
|
||||||
|
docs/examples/employee/ the acceptance workload (Task 6)
|
||||||
|
scripts/employee-accept.sh the gate (Task 6)
|
||||||
|
docs/plan/oop-vm/00-wob-format.md appended builtin ids (Tasks 3-5)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Query expressions parse
|
||||||
|
|
||||||
|
**Concept & reason:** the comprehension grammar from spec section 3 —
|
||||||
|
`from`/`where`/`join`/`group…into`/`order by`/`take`/`skip`/`select`, clause
|
||||||
|
order fixed, aggregates as clause functions — becomes lexer keywords (contextual,
|
||||||
|
so `from`/`group`/`order` stay legal identifiers outside a query), a clause AST,
|
||||||
|
and parse diagnostics in a new WO-E5xx range (clause out of order, missing
|
||||||
|
`select`, aggregate named outside a query). Fixed clause order is a parse rule,
|
||||||
|
not a type rule, so the error lands on the exact token.
|
||||||
|
|
||||||
|
- [ ] Grammar and AST for every clause; contextual keywords verified against
|
||||||
|
the existing samples (no `.wo` file in the tree breaks).
|
||||||
|
- [ ] WO-E5xx diagnostics with golden coverage in the existing `woc-test`
|
||||||
|
suite (dump-ast goldens for well-formed queries, diagnostic goldens for
|
||||||
|
each malformed shape).
|
||||||
|
- [ ] Gates green; commit locally.
|
||||||
|
|
||||||
|
### Task 2: Queries typecheck
|
||||||
|
|
||||||
|
**Concept & reason:** the surface's whole promise is compile-time checking.
|
||||||
|
`from e in Employee` opens a scope where `e`'s fields are the class's; `ref`
|
||||||
|
navigation substitutes the target class's field set (`e.dept.name`);
|
||||||
|
`backlink` reads type as `multi` of the source class; `group … into g` closes
|
||||||
|
the range scope and opens the group scope, where `g.key` has the key's type
|
||||||
|
and `g.f` is legal **only** inside an aggregate call; `select { … }`
|
||||||
|
synthesizes an anonymous record type so later use of a dropped column is a
|
||||||
|
compile error. Aggregate result types follow the spec's table exactly —
|
||||||
|
`count` is Int, `sum` Int, `avg`/`min`/`max` are `?T` because an empty group
|
||||||
|
is data. Unknown table, unknown column (naming the class), navigation through
|
||||||
|
a non-`ref`, bare group-member column, join sides not separable: each is its
|
||||||
|
own WO-E5xx with a golden.
|
||||||
|
|
||||||
|
- [ ] Scope machinery for range/group variables; navigation typing both
|
||||||
|
directions; projection record synthesis interned like other class
|
||||||
|
shapes.
|
||||||
|
- [ ] Aggregate typing per the spec table; `?T` results force the existing
|
||||||
|
nil-handling style at use sites.
|
||||||
|
- [ ] Diagnostic goldens for every error class named above; gates green;
|
||||||
|
commit locally.
|
||||||
|
|
||||||
|
### Task 3: Cursors and integrity in the engine
|
||||||
|
|
||||||
|
**Concept & reason:** the runtime side queries need, built on iteration 9's
|
||||||
|
storage: a table-scan cursor (open by class id, advance, borrowed row view),
|
||||||
|
a primary-index point read (`ref` navigation), a secondary-index cursor with
|
||||||
|
longest-prefix probe (backlink reads, indexed `where`), all as builtins
|
||||||
|
appended to the format doc. Integrity lands at the row choke points where the
|
||||||
|
indexes already live: inserting/updating a non-nil `ref` probes the referenced
|
||||||
|
primary index and traps on a miss (foreign-key violation, sibling of the
|
||||||
|
unique trap); deleting a row still referenced traps (restrict) via the same
|
||||||
|
secondary index a `backlink` requires — one probe, no new structure. Nil
|
||||||
|
`ref` never probes (the MATCH SIMPLE rule); an update that leaves the key
|
||||||
|
unchanged skips the check (both from the PostgreSQL RI survey, mechanism
|
||||||
|
discarded — a check is an index probe, never query text).
|
||||||
|
|
||||||
|
- [ ] Cursor builtins + borrowed-row-view lifetime rules written into the
|
||||||
|
binding doc (a row view never escapes the loop that opened the cursor —
|
||||||
|
the ownership pass enforces it, mirror of the container-read borrow).
|
||||||
|
- [ ] FK trap + restrict trap wired through the row API; a debug-build probe
|
||||||
|
counter exposed for Task 5's index-selection proof.
|
||||||
|
- [ ] db-corpus fixtures from iteration 9's plan extended with one FK-violation
|
||||||
|
and one restrict fixture (trap-code exact); ASan green; commit locally.
|
||||||
|
|
||||||
|
### Task 4: Group hash + aggregate execution
|
||||||
|
|
||||||
|
**Concept & reason:** the `AggregateBy` shape from the spec — group-and-reduce
|
||||||
|
in one pass, no group objects. A group hash table builtin set: create (keyed
|
||||||
|
by the group key's kind, nil legal), upsert-advance (locate-or-create the
|
||||||
|
group, advance each aggregate's transition slot), drain (iterate groups,
|
||||||
|
finalize, hand key + finals to the compiled projection loop). Transition/
|
||||||
|
finalize is PostgreSQL's split: `avg` carries sum+count in its transition
|
||||||
|
state and divides at finalize; the "no row seen yet" state is distinct from
|
||||||
|
"transition value is nil" so nil-skipping aggregates need no first-row special
|
||||||
|
case. Aggregate semantics are the spec's table: empty ⇒ nil (or 0 for
|
||||||
|
count/sum), nil elements skipped, sum wraps, avg truncates.
|
||||||
|
|
||||||
|
- [ ] Builtins implemented; the input projected to only the columns the
|
||||||
|
aggregates read before hashing (the nodeAgg memory lesson).
|
||||||
|
- [ ] Fixture-level verification through iteration 9's db corpus (one grouped
|
||||||
|
query, exact output) plus the sample in Task 6; ASan green; commit
|
||||||
|
locally.
|
||||||
|
|
||||||
|
### Task 5: Lowering — the compiler is the planner
|
||||||
|
|
||||||
|
**Concept & reason:** a query desugars to the bytecode loops the language
|
||||||
|
already has. Scan or probe chosen at compile time: leading `where` equality
|
||||||
|
conjuncts matched against declared indexes longest-prefix-first, probe emitted
|
||||||
|
on a hit, scan otherwise — and the choice is **demonstrated** via Task 3's
|
||||||
|
probe counter in the acceptance, not assumed. `join` builds a transient hash
|
||||||
|
on the inner side and probes with the outer (nil never inserted, never
|
||||||
|
probes). `group` emits the two-phase hash-aggregation loop from Task 4.
|
||||||
|
`order by` materializes and stable-sorts (original-index tiebreak — the LINQ
|
||||||
|
stability guarantee); `take`/`skip` slice the result. Ownership: row views
|
||||||
|
stay loop-bound borrows; everything `select` emits is copied/built at the
|
||||||
|
boundary under the established Text-copy and fresh-value rules — queries add
|
||||||
|
no new ownership classes, and the ownership pass's existing drop machinery
|
||||||
|
covers the query's temporaries because the lowering IS ordinary loops.
|
||||||
|
|
||||||
|
- [ ] Desugar + lowering for every clause; disassembly of the sample's report
|
||||||
|
mode shows loops and builtins, no plan tree, no text.
|
||||||
|
- [ ] Index selection proven: the acceptance asserts probe-counter deltas for
|
||||||
|
the indexed `staff <department>` path versus a full-scan query.
|
||||||
|
- [ ] `oop-e2e`, `woc-test`, ASan-corpus gates green; commit locally.
|
||||||
|
|
||||||
|
### Task 6: The employee sample is the acceptance
|
||||||
|
|
||||||
|
**Concept & reason:** spec section 6 verbatim — `docs/examples/employee` with
|
||||||
|
`Department`/`Employee` (`@table`, `@unique` name, composite `[dept, salary]`
|
||||||
|
index, `ref`/`backlink` pair), wo.toml manifest so `woc .` builds it, a `just`
|
||||||
|
module beside it (the log-watcher convention), and `scripts/employee-accept.sh`
|
||||||
|
as the gate: `seed` (insert + WAL + duplicate-department trap on the second
|
||||||
|
run), `report` (headcount/avg/min/max by department, total payroll, ordered by
|
||||||
|
average salary — byte-exact lines), `staff` (both navigation directions +
|
||||||
|
index-probe proof), `raise` (update through a query; missing department ⇒ the
|
||||||
|
empty-query nil path), the restrict-trap demo, and a kill -9 between `seed`
|
||||||
|
and `report` proving replay on this workload.
|
||||||
|
|
||||||
|
- [ ] Sample written; `woc docs/examples/employee` compiles with zero
|
||||||
|
diagnostics; the binary's modes run.
|
||||||
|
- [ ] `scripts/employee-accept.sh` + `just employee` module: every mode
|
||||||
|
checked with exact expectations, trap codes asserted, crash step
|
||||||
|
included; soak-style RSS/fd sampling reused from the log-watcher
|
||||||
|
script's pattern for the `report` loop.
|
||||||
|
- [ ] ASan run of the full acceptance: zero leaks (the log-watcher bar).
|
||||||
|
- [ ] Docs: README beside the sample, CODE-LOGIC.md updates beside changed
|
||||||
|
compiler/runtime code, format-doc builtin table final, board + story
|
||||||
|
rows updated with measured numbers; commit locally.
|
||||||
|
|
||||||
|
## Out of scope — deferred by name
|
||||||
|
|
||||||
|
- Everything spec section 7 lists: set operators, outer joins, subqueries,
|
||||||
|
composite group keys, groups as values, FK cascade/set-nil, deferred
|
||||||
|
checks, SQL text in any role, cross-shard queries, `LIVE`, migrations,
|
||||||
|
cost-based planning.
|
||||||
|
- Sorted-grouping and partial-sort optimizations (recorded LINQ/postgres
|
||||||
|
precedents; hash + full stable sort are this plan's only strategies).
|
||||||
|
- The ecommerce sample's query rewrite (9b's fifth acceptance criterion) —
|
||||||
|
it lands as its own follow-up once the employee gate is green, so this
|
||||||
|
plan's blast radius stays one new sample.
|
||||||
|
|
@ -2,6 +2,14 @@
|
||||||
|
|
||||||
> A two-layer multi-paradigm language for an e-commerce platform with ACID transactions across relational, document, and graph storage.
|
> A two-layer multi-paradigm language for an e-commerce platform with ACID transactions across relational, document, and graph storage.
|
||||||
|
|
||||||
|
> **Query layer superseded (2026-08-15):** the SQL + Cypher query layer below
|
||||||
|
> is design history — on the C stack, programs query their tables through the
|
||||||
|
> language-integrated surface specified in
|
||||||
|
> [`2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md)
|
||||||
|
> (fork 1's decision record). This document remains the reference for the
|
||||||
|
> schema layer's vocabulary and for the `wo-db` prototype's engine semantics;
|
||||||
|
> SQL text is at most a future export format, never a program surface.
|
||||||
|
|
||||||
**Previous**: [Phase 1 — Database Evaluation](./01-evaluation.md) | **Next**: [Phase 3 — In-Memory Engine](./03-inmemory-engine.md) | **Index**: [database.md](../database.md)
|
**Previous**: [Phase 1 — Database Evaluation](./01-evaluation.md) | **Next**: [Phase 3 — In-Memory Engine](./03-inmemory-engine.md) | **Index**: [database.md](../database.md)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
|
||||||
|
|
@ -8,9 +8,14 @@
|
||||||
> precedes iteration 10 because `service` blocks will want to return query
|
> precedes iteration 10 because `service` blocks will want to return query
|
||||||
> results.
|
> results.
|
||||||
>
|
>
|
||||||
> **No spec exists yet.** This iteration frames the outcome and records the
|
> **Spec exists (2026-08-15):**
|
||||||
> open questions; the design must be brainstormed before a plan is written.
|
> [`2026-08-15-table-relations-query-design.md`](../../superpowers/specs/2026-08-15-table-relations-query-design.md)
|
||||||
> The three questions in *Info* are genuine forks, not details.
|
> settles the three forks recorded in *Info* below (kept as the decision
|
||||||
|
> record): the SQL/Cypher layer is superseded as the program surface,
|
||||||
|
> the syntax is a compiler-desugared comprehension, and the references
|
||||||
|
> contribute vocabulary + semantics (System.Linq) and execution + integrity
|
||||||
|
> vocabulary (PostgreSQL, surveyed with the spec). Plan:
|
||||||
|
> [`2026-08-15-employee-relations-query.md`](../../plan/compiler/2026-08-15-employee-relations-query.md).
|
||||||
|
|
||||||
## Goals
|
## Goals
|
||||||
|
|
||||||
|
|
@ -119,10 +124,18 @@ iteration makes it mean something in the C stack.
|
||||||
|
|
||||||
## Proposed Solution
|
## Proposed Solution
|
||||||
|
|
||||||
- **Brainstorm a spec first**, settling the three forks above; only then write
|
- ~~Brainstorm a spec first~~ — **done 2026-08-15**; the spec settles all
|
||||||
the plan. This iteration deliberately ships no plan pointer, because
|
three forks and the plan exists (pointers in the header note). The fork-1
|
||||||
choosing between "replace the SQL layer" and "sit beside it" changes what
|
outcome for the record: language-integrated query is the only program
|
||||||
the plan contains.
|
surface; `docs/runtime/database/02-wo-language.md`'s SQL/Cypher layer stays
|
||||||
|
as design history and as the `wo-db` prototype's engine-semantics
|
||||||
|
reference, never as syntax.
|
||||||
|
- **The acceptance workload is a new sample**: `docs/examples/employee` —
|
||||||
|
`Department`/`Employee` with `@unique`, a composite index, a `ref`/
|
||||||
|
`backlink` pair, and a report mode that is one `GROUP BY` after another
|
||||||
|
(headcount, avg/min/max salary by department). It is 9b's acceptance the
|
||||||
|
way log-watcher was iterations 1–7's; the ecommerce query rewrite (the
|
||||||
|
fifth criterion below) follows as its own step once employee is green.
|
||||||
- Study `.dev/reference/dotnet-runtime`'s `System.Linq` operator set for the
|
- Study `.dev/reference/dotnet-runtime`'s `System.Linq` operator set for the
|
||||||
vocabulary, and `docs/runtime/database/02-wo-language.md` plus
|
vocabulary, and `docs/runtime/database/02-wo-language.md` plus
|
||||||
`prototypes/wo-db/` for the semantics already committed to.
|
`prototypes/wo-db/` for the semantics already committed to.
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,307 @@
|
||||||
|
# `@table`, Relations, and Language-Integrated Query — Design
|
||||||
|
|
||||||
|
> **Status: proposed** (story iteration 9b). Settles the three forks recorded in
|
||||||
|
> [`09b-table-relations-query.md`](../../stories/language-runtime-database/09b-table-relations-query.md).
|
||||||
|
> Plan: [`docs/plan/compiler/2026-08-15-employee-relations-query.md`](../../plan/compiler/2026-08-15-employee-relations-query.md).
|
||||||
|
> Depends on iteration 9's engine plan
|
||||||
|
> ([`2026-08-01-db-engine-binding.md`](../plans/2026-08-01-db-engine-binding.md))
|
||||||
|
> for storage, WAL, insert and the select subset.
|
||||||
|
|
||||||
|
**One sentence:** queries are written in the language, checked by the compiler,
|
||||||
|
and lowered to bytecode loops over a handful of engine cursor builtins — no SQL
|
||||||
|
text exists anywhere in a compiled program — and the acceptance workload is a
|
||||||
|
new `docs/examples/employee` sample whose report mode is one `GROUP BY` after
|
||||||
|
another.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The three forks, settled
|
||||||
|
|
||||||
|
### Fork 1 — the SQL/Cypher layer is superseded as the program surface
|
||||||
|
|
||||||
|
`docs/runtime/database/02-wo-language.md` specifies a query layer of literal
|
||||||
|
SQL and Cypher with five fixed-glue rules. That layer is **not built on the C
|
||||||
|
stack**. The language-integrated surface below is the only way a `.wo` program
|
||||||
|
queries its tables.
|
||||||
|
|
||||||
|
Reasons, in order of weight:
|
||||||
|
|
||||||
|
- The iteration's own acceptance forbids the alternative: a query must lower to
|
||||||
|
engine operations "not a string handed to a parser at runtime — provable by
|
||||||
|
disassembly." A resident SQL parser and a compile-checked surface would be
|
||||||
|
two grammars for one meaning, and the second grammar would re-introduce at
|
||||||
|
runtime every error class the first one eliminated at compile time.
|
||||||
|
- The language's identity is compile-time checking (the WO-E diagnostic
|
||||||
|
catalogue). "A typo in a field name is a compile error" cannot be delivered
|
||||||
|
by a text layer.
|
||||||
|
- A binary that ships a SQL parser it uses only for its own programs pays
|
||||||
|
image size and attack surface for nothing.
|
||||||
|
|
||||||
|
What survives: `02-wo-language.md` stays as design history and as the
|
||||||
|
specification the `prototypes/wo-db` C++ prototype implements; that prototype
|
||||||
|
remains the **engine-semantics** reference (what an index probe returns, what
|
||||||
|
unique violation means), not a syntax reference. SQL text remains a candidate
|
||||||
|
*export/interop* format for much later (external tools speaking to a writeonce
|
||||||
|
service), explicitly not part of this iteration.
|
||||||
|
|
||||||
|
### Fork 2 — comprehension syntax, desugared at compile time
|
||||||
|
|
||||||
|
writeonce has no function values, so LINQ's method-chains-taking-lambdas are
|
||||||
|
unwritable. The surface is a **query expression** — a comprehension the
|
||||||
|
compiler desugars — where every predicate and projection is ordinary
|
||||||
|
expression syntax with the range variable in scope:
|
||||||
|
|
||||||
|
```wo
|
||||||
|
let seniors = from e in Employee
|
||||||
|
where e.salary > 90000
|
||||||
|
order by e.salary desc
|
||||||
|
select e;
|
||||||
|
|
||||||
|
let by_dept = from e in Employee
|
||||||
|
group e by e.dept into g
|
||||||
|
select { dept: g.key.name, headcount: count(g), avg_salary: avg(g.salary) };
|
||||||
|
```
|
||||||
|
|
||||||
|
(Illustrative; the grammar is normative in prose, section 3.)
|
||||||
|
|
||||||
|
Why comprehension and not chaining: a chained `.where(e.salary > 100)` has no
|
||||||
|
binding site for `e` — the comprehension's `from e in` clause is what
|
||||||
|
introduces the variable, which is exactly the property that makes every later
|
||||||
|
clause an ordinary typed expression the existing typechecker can check. C# had
|
||||||
|
to add query expressions *on top of* lambdas for the same readability reason;
|
||||||
|
we get to skip the lambda layer entirely. The predicate is compile-time
|
||||||
|
syntax, never a runtime closure — which is also what lets the whole query
|
||||||
|
lower to plain bytecode.
|
||||||
|
|
||||||
|
### Fork 3 — what each reference contributes
|
||||||
|
|
||||||
|
**System.Linq** (surveyed 2026-08-15) contributes the operator vocabulary and
|
||||||
|
its semantic edge cases, not machinery:
|
||||||
|
|
||||||
|
- The lowering target for `group … select aggregate` is the shape of
|
||||||
|
`AggregateBy`/`CountBy` and the `GroupBy(key, resultSelector)` overload:
|
||||||
|
**group-and-reduce as one node, no intermediate group object materialized**
|
||||||
|
(`Grouping.cs:63`, `AggregateBy.cs`). The `IGrouping`-returning overloads
|
||||||
|
exist to hand groups around as values; we have no delegates to hand them to,
|
||||||
|
so groups surface only as the `into g` binding inside the query itself.
|
||||||
|
- Empty-source rules per aggregate (section 4's table) adapt LINQ's — where
|
||||||
|
C# throws `InvalidOperationException`, writeonce answers `nil` through a
|
||||||
|
`?T` result, because an empty group is data, not a fault.
|
||||||
|
- Ordering is **stable**, guaranteed — LINQ enforces it with an original-index
|
||||||
|
tiebreak (`OrderedEnumerable.cs:432`); we adopt the same guarantee and the
|
||||||
|
same escape hatch (an unstable sort is legal when the key is a scalar and
|
||||||
|
rows are not identity-bearing, `OrderBy.cs:144-163` precedent).
|
||||||
|
- Join semantics: build a hash on the inner side, probe with the outer, and
|
||||||
|
**nil never joins** — LINQ achieves SQL's NULL-never-matches by refusing to
|
||||||
|
insert null keys into the build side (`Lookup.cs:119-122`); we do the same.
|
||||||
|
In grouping, by contrast, nil **is** a legitimate key (`Lookup.cs:207`).
|
||||||
|
|
||||||
|
**PostgreSQL** (surveyed 2026-08-15) contributes execution vocabulary for the
|
||||||
|
engine side:
|
||||||
|
|
||||||
|
- Aggregate execution is the transition/finalize split from `nodeAgg.c`: a
|
||||||
|
per-group transition value advanced once per row, a finalize step converting
|
||||||
|
state to result (`avg` carries sum+count without the executor knowing). The
|
||||||
|
`noTransValue` vs `transValueIsNull` distinction — "no row seen yet" is not
|
||||||
|
"the value is nil" — is adopted verbatim; it is what makes nil-skipping
|
||||||
|
aggregates correct without special-casing the first row.
|
||||||
|
- Grouping strategy: hash aggregation (`AGG_HASHED`) is the only strategy this
|
||||||
|
iteration builds; sorted grouping is an optimization for later. Project the
|
||||||
|
input to only the columns the aggregate needs before hashing
|
||||||
|
(`find_hash_columns` precedent).
|
||||||
|
- Referential integrity semantics from `ri_triggers.c`, mechanism discarded:
|
||||||
|
an FK check is a **direct probe of the referenced table's primary index**
|
||||||
|
(never query text); a nil `ref` passes the check (MATCH SIMPLE rule); an
|
||||||
|
update that does not change the key skips the check
|
||||||
|
(`RI_FKey_fk_upd_check_required` precedent). Enforcement action is
|
||||||
|
**restrict only** — deleting a Department that still has Employees traps;
|
||||||
|
cascade/set-nil are out of scope.
|
||||||
|
- Everything MVCC, buffer-manager, lock-manager and cost-planner shaped is
|
||||||
|
explicitly non-transferable: single-writer-per-shard RAM-authoritative
|
||||||
|
storage designed those problems away (iteration 9's doctrine).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Relations become typed navigation
|
||||||
|
|
||||||
|
The declaration vocabulary already exists in the samples and partially in the
|
||||||
|
compiler; this iteration makes it mean something:
|
||||||
|
|
||||||
|
- `dept: ref Department` — a foreign key. Stored as the target's row id (a
|
||||||
|
scalar column, already the compiler's classification). **Navigating** it in
|
||||||
|
an expression (`e.dept.name`) typechecks with `Department`'s field set in
|
||||||
|
scope and lowers to a primary-index point read.
|
||||||
|
- `staff: backlink Employee.dept` — the declared inverse. Not a stored column;
|
||||||
|
reading it (`d.staff`) is a secondary-index scan of `Employee`'s `dept`
|
||||||
|
column and yields `multi Employee`. Declaring a `backlink` whose target
|
||||||
|
field is not a `ref` to this class is a compile error.
|
||||||
|
- `?ref Department` — an optional relation; nil stores as id 0, never probes,
|
||||||
|
never joins.
|
||||||
|
|
||||||
|
Integrity, enforced at the engine's row choke points (iteration 9's index
|
||||||
|
doctrine — nothing touches storage except the row API):
|
||||||
|
|
||||||
|
- Insert/update of a non-nil `ref`: probe the referenced primary index; a miss
|
||||||
|
traps with a foreign-key violation code (sibling of iteration 9's
|
||||||
|
unique-violation trap).
|
||||||
|
- Delete of a row that a non-nil `ref` still points at: trap (restrict). The
|
||||||
|
check is a probe of the secondary index that the `backlink` already
|
||||||
|
requires, so restrict costs one lookup and no new structure.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. The query surface (normative, in prose)
|
||||||
|
|
||||||
|
A **query expression** is an expression. Its clauses, in the only order they
|
||||||
|
may appear:
|
||||||
|
|
||||||
|
1. `from <var> in <source>` — required, first. Source is a table (class name),
|
||||||
|
a `backlink` navigation, or a `multi` value. Introduces the range variable.
|
||||||
|
2. `where <bool-expr>` — optional, repeatable. Ordinary boolean expression
|
||||||
|
over range variables; each `where` is a filter.
|
||||||
|
3. `join <var2> in <source2> on <expr> == <expr2>` — optional. Equi-join only;
|
||||||
|
one side references only the outer variable, the other only the joined one.
|
||||||
|
Most joins in practice are spelled as `ref` navigation instead; explicit
|
||||||
|
`join` exists for joining on non-relation columns.
|
||||||
|
4. `group <expr> by <key-expr> into <g>` — optional. Ends the scope of the
|
||||||
|
range variables and opens the scope of `g`. `g.key` is the key's value.
|
||||||
|
For any field `f` of the grouped element, `g.f` names the **column of
|
||||||
|
members** — legal only inside an aggregate call. Nil is a legitimate key.
|
||||||
|
5. `order by <expr> [desc]` — optional, comma-repeatable keys. Stable.
|
||||||
|
6. `take <int-expr>` / `skip <int-expr>` — optional.
|
||||||
|
7. `select <expr>` — required, last. The result element: a whole row, a single
|
||||||
|
field, or a projection literal `{ name: expr, … }` whose type the compiler
|
||||||
|
synthesizes as an anonymous record. Using a column the projection dropped
|
||||||
|
is a compile error thereafter (the 9b acceptance's projection criterion).
|
||||||
|
|
||||||
|
A query's value is `multi <element>`; a query wrapped directly in a whole-
|
||||||
|
query aggregate (`count(from …)`) is that aggregate's scalar. Execution is
|
||||||
|
eager at the point the expression is evaluated — no deferred queries, no query
|
||||||
|
values passed around (that would be a function value in disguise).
|
||||||
|
|
||||||
|
Aggregates are compiler-recognized **clause functions**, legal over a group
|
||||||
|
binding or a whole query: `count(g)`, `count(from …)`, `sum(g.f)`, `avg(g.f)`,
|
||||||
|
`min(g.f)`, `max(g.f)`. They are not general functions; naming one outside a
|
||||||
|
query is the existing unknown-identifier error.
|
||||||
|
|
||||||
|
Diagnostics this surface owns (new WO-E5xx range): unknown table, unknown
|
||||||
|
column (naming the class), relation navigated through a non-`ref` field,
|
||||||
|
aggregate outside a query, group member column used bare outside an aggregate,
|
||||||
|
join sides not separable, projection field name collision.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Aggregate semantics (normative table)
|
||||||
|
|
||||||
|
| Aggregate | Input | Result type | Empty source | Nil elements (`?T` column) |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `count(g)` | group/query | `Int` | `0` | counted (row exists) |
|
||||||
|
| `sum(g.f)` | `Int` column | `Int` | `0` | skipped |
|
||||||
|
| `avg(g.f)` | `Int` column | `?Int` | `nil` | skipped; all-nil ⇒ `nil` |
|
||||||
|
| `min(g.f)` / `max(g.f)` | `Int` or `Text` column | `?T` | `nil` | skipped; all-nil ⇒ `nil` |
|
||||||
|
|
||||||
|
Decisions behind the table:
|
||||||
|
|
||||||
|
- **Empty is data, not a fault.** LINQ throws on empty `Average`/`Min`/`Max`
|
||||||
|
over non-nullables; writeonce has `?T` and a nil-forcing style already, so
|
||||||
|
the nullable-column LINQ behavior (`null` on empty, skip nils, all-nil ⇒
|
||||||
|
`null` — `Average.cs:165`, `Min.cs:47-56`) is the behavior for everyone.
|
||||||
|
Traps stay reserved for integrity violations.
|
||||||
|
- **`sum` wraps.** The VM's ADD wraps two's-complement; `sum` is a loop of
|
||||||
|
ADDs and inherits that. Documented, consistent with language arithmetic,
|
||||||
|
and the alternative (checked overflow, LINQ's `Sum.cs:40`) would make `sum`
|
||||||
|
the only trapping arithmetic in the language.
|
||||||
|
- **`avg` truncates** toward zero (integer division), same as `/`.
|
||||||
|
- Execution is the transition/finalize ABI (section 1, fork 3): one opaque
|
||||||
|
transition slot per (group, aggregate), advanced per row, finalized per
|
||||||
|
group when the hash table drains.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Lowering and execution model
|
||||||
|
|
||||||
|
A query compiles to an **ordinary bytecode loop** — the same opcodes every
|
||||||
|
`for` loop uses — over a small set of new engine cursor builtins. There is no
|
||||||
|
plan-tree interpreter in the VM: the compiler *is* the planner, at the only
|
||||||
|
scale this iteration promises (index selection, no cost model).
|
||||||
|
|
||||||
|
- **Scan**: table cursor builtins (open by class id, advance, read row into a
|
||||||
|
register as a borrowed row view). `where` clauses are ordinary compiled
|
||||||
|
boolean expressions guarding the loop body — the flattened-expression
|
||||||
|
lesson of PostgreSQL's executor (`execExpr.c`) is in our case simply "the
|
||||||
|
bytecode we already generate."
|
||||||
|
- **Index selection**: if the leading `where` conjuncts equality-match a
|
||||||
|
declared index's prefix (longest-prefix rule, iteration 9's `find_by`), the
|
||||||
|
compiler emits an index-probe cursor instead of a scan, and the acceptance
|
||||||
|
demonstrates the difference (a probe counter the engine exposes in debug
|
||||||
|
builds — "demonstrated, not assumed").
|
||||||
|
- **`ref` navigation** lowers to a primary-index point-read builtin.
|
||||||
|
**`backlink`** lowers to a secondary-index cursor over the ref column.
|
||||||
|
- **`group … into g … select`** lowers to the two-phase hash-aggregation
|
||||||
|
shape: phase one iterates the source, keying a group hash table and
|
||||||
|
advancing transition slots (builtins: group-table create / upsert-advance);
|
||||||
|
phase two drains the table, runs finalize, evaluates the `select`
|
||||||
|
projection per group. One node, no intermediate group objects — the
|
||||||
|
`AggregateBy` shape.
|
||||||
|
- **`join`** lowers to build-inner/probe-outer against a transient hash
|
||||||
|
keyed on the join column; nil keys are never inserted and never probe.
|
||||||
|
- **`order by`** materializes the result `multi` and stable-sorts it; `take`/
|
||||||
|
`skip` slice afterwards (a partial-sort fast path is recorded as a later
|
||||||
|
optimization, LINQ `SpeedOpt` precedent).
|
||||||
|
- **Ownership**: rows read from a table are engine-owned; anything a query
|
||||||
|
*returns* is copied out at the select boundary under Task-2's established
|
||||||
|
rule — a Text crossing an ownership boundary is copied, a projection record
|
||||||
|
is freshly built and caller-owned. Queries introduce no new ownership
|
||||||
|
classes.
|
||||||
|
|
||||||
|
Format consequences: new builtin ids appended to the format doc's table (the
|
||||||
|
cursor/group/probe set), no new opcodes, no version bump beyond iteration 9's.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Acceptance workload — `docs/examples/employee`
|
||||||
|
|
||||||
|
A new sample, structured like `log-watcher` (wo.toml manifest, program mode,
|
||||||
|
`just` module, acceptance script), small enough to read in one sitting and
|
||||||
|
shaped so every 9b feature is load-bearing:
|
||||||
|
|
||||||
|
- **Types**: `Department` (`@table(name: "departments", index: [name])`,
|
||||||
|
`name: Text @unique`) and `Employee` (`@table(name: "employees",
|
||||||
|
index: [dept], index: [dept, salary])`, `name: Text`, `salary: Int` cents,
|
||||||
|
`hired: Int` ms, `dept: ref Department`); `Department.staff: backlink
|
||||||
|
Employee.dept`.
|
||||||
|
- **Modes** (CLI, exit codes per the program-mode contract):
|
||||||
|
- `seed` — inserts departments and employees; proves insert + WAL + unique
|
||||||
|
trap (second `seed` run reports the duplicate-department trap caught).
|
||||||
|
- `report` — the GroupBy showcase: headcount by department, average /
|
||||||
|
min / max salary by department, total payroll, departments ordered by
|
||||||
|
average salary — each line's expected output is byte-exact in the
|
||||||
|
acceptance script.
|
||||||
|
- `staff <department>` — relation navigation both directions: finds the
|
||||||
|
department by unique name (index probe), lists its `staff` backlink
|
||||||
|
ordered by salary; also prints each employee's `e.dept.name` to prove
|
||||||
|
forward navigation.
|
||||||
|
- `raise <department> <pct>` — update through a query result; re-running
|
||||||
|
`report` shows the moved averages; a `raise` for a missing department
|
||||||
|
exercises the empty-query path (`avg` ⇒ nil).
|
||||||
|
- **Integrity demo**: deleting a department with staff traps (restrict);
|
||||||
|
the acceptance asserts the trap code.
|
||||||
|
- **Crash step**: kill -9 between `seed` and `report`, re-run `report` —
|
||||||
|
iteration 9's WAL replay proven on this workload too.
|
||||||
|
|
||||||
|
The sample is 9b's acceptance the way log-watcher was iterations 1–7's: no new
|
||||||
|
corpus fixtures beyond the db corpus iteration 9 already plans; the sample is
|
||||||
|
the test.
|
||||||
|
|
||||||
|
## 7. Out of scope (inherited and new)
|
||||||
|
|
||||||
|
- Everything 9b's story already excludes: cross-shard queries and distributed
|
||||||
|
joins, `LIVE` subscriptions, migrations, cost-based planning.
|
||||||
|
- The `IGrouping`-as-value surface (groups escaping their query), `Distinct`/
|
||||||
|
set operators, outer joins (`LeftJoin` family), subqueries in `where`, and
|
||||||
|
`group by` composite keys beyond a single expression — each waits for a
|
||||||
|
workload that demands it.
|
||||||
|
- FK actions other than restrict (cascade, set-nil), deferred constraint
|
||||||
|
checking (the after-trigger queue pattern is recorded for when transactions
|
||||||
|
span statements).
|
||||||
|
- SQL text in any runtime role.
|
||||||
Loading…
Reference in a new issue