From 43f0082c7108db696ddf1857b458aad72e6f7d4d Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Sun, 16 Aug 2026 19:28:25 +0200 Subject: [PATCH] docs: iteration 9g story -- query grammar from real embedded-DB corpora MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - method: an embedded-SQL app is a grammar corpus; catalogue what it actually uses and add only that, translating its statements 1:1 as the acceptance (the postgres/System.Linq reference pattern applied to a whole application) - corpus #1 = ~/projects/skillhost (C++ MCP host, embedded SQLite skill catalog): surveyed, entire SQL footprint is one file β€” 1 table, a single-row parameterized INSERT, 4 SELECTs. 3 of 5 statements already run on the 9b surface (insert, where name==?, order by name; PK β‰ˆ @unique). Exactly 2 are the real gap: - whole-query `count` (group-free; the degenerate aggregate, NOT the parked group-by) - correlated `not exists` subquery (skillhost's roots-of-the-tree) - notable finding: skillhost's NOT EXISTS is naturally a `backlink` emptiness in writeonce (children backlink + len==0), so the corpus may be fully expressible once len(query) is confirmed β€” the iteration may collapse to "confirm len(query) + add exists"; forks record this - explicitly parks everything skillhost does NOT use (join/having/ offset/distinct/CTE/window/union/upsert/returning/json/fts/triggers) and the full group-by; each enters only when a corpus demands it - acceptance: docs/examples/skill-catalog mirroring skillhost's schema + its 5 catalog ops as writeonce translations - roadmap + board rows added Co-Authored-By: Claude Opus 5 (1M context) --- docs/00-status.md | 2 + .../language-runtime-database/00-story.md | 1 + .../09g-query-grammar-corpus.md | 161 ++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 docs/stories/language-runtime-database/09g-query-grammar-corpus.md diff --git a/docs/00-status.md b/docs/00-status.md index 2686d39..5ce312f 100644 --- a/docs/00-status.md +++ b/docs/00-status.md @@ -121,6 +121,7 @@ that sequences its tasks. Read one, approve, then the next starts. | 9d | [Keypair attach auth](stories/language-runtime-database/09d-keypair-attach-auth.md) | πŸ”„ crypto+handshake done (branch keypair-auth); manifest pending | | 9e | [Durability, throughput, scale](stories/language-runtime-database/09e-durability-throughput-scale.md) | ⬜ needs a spec first | | 9f | [io_uring group-commit](stories/language-runtime-database/09f-io-uring-commit.md) | ⬜ after 8 + 9e | +| 9g | [Query grammar corpus](stories/language-runtime-database/09g-query-grammar-corpus.md) | ⬜ needs a spec first | | 10 | [HTTP service layer](stories/language-runtime-database/10-http-service.md) | ⬜ | Hold | | 11 | [Fibers](stories/language-runtime-database/11-fibers.md) | ⬜ | Hold | | 12 | [Blue-green deploy](stories/language-runtime-database/12-blue-green-deploy.md) | ⬜ | Hold | @@ -332,6 +333,7 @@ Ecommerce sample (verified 2026-06-13): `api.rest` 17/17 expected statuses pass. | 9d | Keypair attach auth β€” mutual challenge–response, grants name public keys, uid superseded | **no spec yet** β€” four forks recorded; plan folds into 9c's | | 9e | Durability + throughput + scale β€” restart-persistence, read/write benchmark, ~1M rows; the gate every later optimization re-runs | **no spec yet** β€” four forks recorded; the measurement backbone | | 9f | io_uring group-commit write path β€” batched durability overlapped on shard threads, fsync fallback | **no spec yet** β€” brainstorm after iterations 8 + 9e | +| 9g | Query grammar from real embedded-DB corpora β€” whole-query count + correlated exists, driven by the skillhost SQL catalogue; add only what a corpus uses | **no spec yet** β€” three forks; may collapse to "confirm len(query) + add exists" | | 10 | HTTP service layer | [plan 6](superpowers/plans/2026-08-01-http-service-layer.md) | | 11 | Fibers | vision Β§3, [blue-green exploration](plan/exploration/blue-green-vm/00-vision.md) | | 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) β€” plan authored after iterations 9–10 | diff --git a/docs/stories/language-runtime-database/00-story.md b/docs/stories/language-runtime-database/00-story.md index 96fcbb8..4f896b3 100644 --- a/docs/stories/language-runtime-database/00-story.md +++ b/docs/stories/language-runtime-database/00-story.md @@ -57,6 +57,7 @@ iterations); no commits by agents β€” drafts go to `.dev/commit.md`. | 9d | [Keypair attach auth](09d-keypair-attach-auth.md) | program identity is a keypair: mutual challenge–response at attach, grants name public keys, replay-proof, rotation is a config change | | 9e | [Durability, throughput, scale](09e-durability-throughput-scale.md) | restart-persistence proof, read/write benchmark, ~1M-row load; the measurement gate every optimization signs | | 9f | [io_uring group-commit](09f-io-uring-commit.md) | replace fsync-per-commit with io_uring batched durability, overlapped on the shard threads; fsync fallback kept | +| 9g | [Query grammar corpus](09g-query-grammar-corpus.md) | grow the query grammar from real embedded-DB apps: `count`/existence subqueries from the skillhost corpus; add only what a corpus uses | | 10 | [HTTP service layer](10-http-service.md) | `service` blocks route to VM methods; REST parity with Stage 2 | | 11 | [Fibers](11-fibers.md) | green threads on the shard scheduler: reduction-budget preemption, park on I/O | | 12 | [Blue-green deploy](12-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback | diff --git a/docs/stories/language-runtime-database/09g-query-grammar-corpus.md b/docs/stories/language-runtime-database/09g-query-grammar-corpus.md new file mode 100644 index 0000000..25a3fcb --- /dev/null +++ b/docs/stories/language-runtime-database/09g-query-grammar-corpus.md @@ -0,0 +1,161 @@ +# Iteration 9g β€” query grammar, driven by real embedded-DB corpora + +> Format: fiberloom `product/story-iteration-template`. Part of +> [Story β€” one language, one runtime, one database, one binary](00-story.md). +> +> **Inserted 2026-08-16.** A query-surface iteration in the 9b family: the +> language-integrated query grows to cover the grammar that *real +> applications backed by an embedded SQL database actually use* β€” measured, +> not guessed, by cataloguing a real app and adding only the constructs it +> depends on. The method is the postgres/System.Linq reference pattern +> applied to a whole application: **an embedded-SQLite app is a grammar +> corpus; each one analysed drives a grammar increment.** +> +> **No spec exists yet.** The forks in *Info* are genuine decisions. + +## Why this iteration exists + +The 9b query surface ships scan / `where` / `select` / `order by` / `take`, +`ref`/`backlink` navigation, insert / update / delete, `@unique`, and FK +restrict (all running, `docs/examples/employee`). What it does NOT yet cover +is everything past that β€” and "everything" is unbounded, so the sensible way +to choose the *next* grammar is to point at a real program that uses an +embedded SQL database and add exactly what it needs. + +**Corpus #1: `~/projects/skillhost`** (a C++ MCP host, embedded SQLite as its +in-memory skill catalog; surveyed 2026-08-16). Its entire SQL footprint is +one file (`src/catalog/catalog.cpp`, 172 lines): one table + index, a +single-row parameterized `INSERT`, and four `SELECT`s. Mapping each statement +to the writeonce query surface: + +| skillhost statement | writeonce today | +| --- | --- | +| `INSERT INTO skills (…) VALUES (?,…)` | βœ… `insert Skill { … }` | +| `SELECT … WHERE name = ?` | βœ… `from s in Skill where s.name == n select s` (unique-index probe) | +| `SELECT … ORDER BY name` | βœ… `from s in Skill order by s.name select s` | +| `SELECT COUNT(*) FROM skills` | ❌ **whole-query `count`** | +| `SELECT … WHERE NOT EXISTS (SELECT 1 FROM skills c WHERE c.parent = s.name) ORDER BY name` | ❌ **correlated `not exists` subquery** | + +So the real grammar gap this corpus demands is **two constructs**, and β€” +importantly β€” *neither is the parked full group-by/projection machinery*. +Everything else SQLite offers (JOIN, HAVING, LIMIT/OFFSET, DISTINCT, CTE, +window functions, UNION, UPSERT, RETURNING, JSON1, FTS5, triggers, generated +columns) skillhost does not touch, so none of it is in scope here. + +## Goals + +- **Whole-query `count`**: `count(from s in Table [where …] select …)` yields + an `Int` β€” the trivial, group-free special case of aggregation (materialize + the query, take its length). It is a stepping stone toward, and independent + of, the parked group-by aggregation. +- **Existence subqueries**: `exists()` and `not exists()` as a + boolean, usable in a `where` guard, where the inner query may reference the + outer range variable (a *correlated* subquery β€” skillhost's roots-of-the- + tree query). Short-circuits: existence needs only the first matching row. +- **Parity, proven by translation**: a new `docs/examples/skill-catalog` + sample mirrors skillhost's schema and expresses all five of its statements + in writeonce, producing results identical to what skillhost's SQLite + returns for the same data. + +## Acceptance Criteria + +- What to achieve? + - **Given** `count(from s in Skill select s)` and + `count(from s in Skill where s.parent == nil select s)`, + - **when** compiled and run, + - **then** each yields the correct row count as an `Int`, lowered to a + materialize-then-length over the existing scan/where loop β€” no group + machinery, provable by disassembly. +- What to achieve? + - **Given** `from s in Skill where not exists(from c in Skill where + c.parent == s.name select c) order by s.name select s` β€” the roots of + the skill tree, + - **when** run over a catalog with parent/child skills, + - **then** it returns exactly the childless skills in name order, and the + inner query correctly sees the outer `s` (correlation), matching + skillhost's `NOT EXISTS` result row-for-row. +- What to achieve? + - **Given** the `skill-catalog` sample seeded with the same rows a + skillhost session would load, + - **when** each of skillhost's five catalog operations is run through the + writeonce translation, + - **then** every result matches, and the sample's README records the one + translation choice made (see fork 1). + +## Out Of Scope + +- **Full group-by aggregation** (`group … by … into g … select { count(g), + avg(g.f) }`) β€” still parked (9b's deferral). Whole-query `count` here is the + degenerate case, not the general one; `sum`/`avg`/`min`/`max` as query + aggregates ride with the group-by iteration. +- **Every SQL construct skillhost does not use**: JOIN, HAVING, LIMIT/OFFSET + (writeonce has `take`; `skip`/offset waits for a workload), DISTINCT, CTE / + `WITH RECURSIVE`, window functions, UNION/INTERSECT/EXCEPT, UPSERT / + `ON CONFLICT`, `RETURNING`, multi-row `VALUES`, `INSERT … SELECT`, JSON1 + operators, FTS5, triggers, generated columns, explicit collation. Each + enters only when a corpus demands it β€” that is this iteration's whole + method. +- **A resident SQL parser** β€” the doctrine stands: skillhost is a grammar + *corpus* to translate against, never a syntax writeonce adopts. No SQL text + in a compiled image. +- **Subqueries in general** beyond correlated `exists`/`not exists` (e.g. a + subquery producing a value, `IN (subquery)`, scalar subqueries) β€” add when + a corpus uses them. + +## Info + +Forks the spec must settle: + +**1. Does skillhost's `NOT EXISTS` even need a subquery, or does a `backlink` +express it?** skillhost's `skills` table is self-referential (`parent` β†’ a +`name`), and its roots query is "skills no other skill names as parent." In +writeonce that is naturally a **backlink emptiness**: give `Skill` a +`children: backlink Skill.parent` and write `where len(s.children) == 0` β€” no +subquery at all, using machinery that already exists (backlink probe) plus a +`len` on the result. So the corpus may be fully expressible *today* once +`count`/`len` over a query lands, making `exists` strictly optional for +skillhost. Leaning: ship whole-query `count`/`len` (needed regardless), and +add `exists`/`not exists` as the general construct for correlations a backlink +cannot express (a correlation on a non-relation column) β€” but let the +`skill-catalog` sample use the idiomatic backlink form for its roots query and +record the subquery form as the alternative. This keeps the new surface +minimal and honest about what the corpus actually forces. + +**2. `count` vs `len`.** writeonce already has `len`/`count` builtins on a +`multi`. A query yields a `multi`, so `len(from … select …)` may already work +with no new surface at all β€” the "gap" could be purely that a bare query in +argument position typechecks and lowers. Leaning: verify `len()` works +end to end first; if it does, whole-query `count` is a documentation/alias +matter, not new code, and the only real new construct in this iteration is the +existence subquery (fork 1's optional half). The spec must confirm this +against the running compiler before committing scope. + +**3. Correlated-subquery execution.** If `exists` lands, the inner query +references the outer row, so it re-evaluates per outer row (a nested loop) or +uses the referenced index. Leaning: nested-loop for correctness first (the +data is small; skillhost's catalog is dozens of skills), index-backed probe as +the optimization the spec records β€” mirroring how 9b did scans before index +selection. + +**Method note (the durable part):** this iteration establishes the pattern for +*all* future query-grammar growth β€” catalogue a real embedded-DB application, +add only the constructs it uses, translate its statements 1:1 as the +acceptance, and park the rest by name. skillhost is corpus #1 and, tellingly, +needs almost nothing beyond what 9b already shipped β€” which is itself the +strongest evidence that the 9b surface was scoped right. + +## Proposed Solution + +- **Brainstorm the spec** settling the three forks β€” especially forks 1/2, + which may collapse the iteration to "confirm `len()` works + add + `exists`," a very small increment. +- **Acceptance workload**: `docs/examples/skill-catalog` β€” `Skill { name: + Text @unique, description: Text, location: Text, root: Text, parent: ?ref + Skill, children: backlink Skill.parent }` and a CLI mirroring skillhost's + catalog operations (add, get-by-name, list, list-roots, count), each a + direct translation of the corresponding SQLite statement, with an + acceptance script asserting the same results skillhost produces. +- Expected shape: small parser/type/emit additions for `exists`/`not exists` + (a query in boolean position, correlated), whole-query `count`/`len` + confirmed or wired, and the sample + script. No engine format change beyond + what 9b already appended.