docs: iteration 9g story -- query grammar from real embedded-DB corpora
- method: an embedded-SQL app is a grammar corpus; catalogue what it actually uses and add only that, translating its statements 1:1 as the acceptance (the postgres/System.Linq reference pattern applied to a whole application) - corpus #1 = ~/projects/skillhost (C++ MCP host, embedded SQLite skill catalog): surveyed, entire SQL footprint is one file — 1 table, a single-row parameterized INSERT, 4 SELECTs. 3 of 5 statements already run on the 9b surface (insert, where name==?, order by name; PK ≈ @unique). Exactly 2 are the real gap: - whole-query `count` (group-free; the degenerate aggregate, NOT the parked group-by) - correlated `not exists` subquery (skillhost's roots-of-the-tree) - notable finding: skillhost's NOT EXISTS is naturally a `backlink` emptiness in writeonce (children backlink + len==0), so the corpus may be fully expressible once len(query) is confirmed — the iteration may collapse to "confirm len(query) + add exists"; forks record this - explicitly parks everything skillhost does NOT use (join/having/ offset/distinct/CTE/window/union/upsert/returning/json/fts/triggers) and the full group-by; each enters only when a corpus demands it - acceptance: docs/examples/skill-catalog mirroring skillhost's schema + its 5 catalog ops as writeonce translations - roadmap + board rows added Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
0a8ca03ccd
commit
86e1fd4a4e
3 changed files with 164 additions and 0 deletions
|
|
@ -121,6 +121,7 @@ that sequences its tasks. Read one, approve, then the next starts.
|
|||
| 9d | [Keypair attach auth](stories/language-runtime-database/09d-keypair-attach-auth.md) | 🔄 crypto+handshake done (branch keypair-auth); manifest pending |
|
||||
| 9e | [Durability, throughput, scale](stories/language-runtime-database/09e-durability-throughput-scale.md) | ⬜ needs a spec first |
|
||||
| 9f | [io_uring group-commit](stories/language-runtime-database/09f-io-uring-commit.md) | ⬜ after 8 + 9e |
|
||||
| 9g | [Query grammar corpus](stories/language-runtime-database/09g-query-grammar-corpus.md) | ⬜ needs a spec first |
|
||||
| 10 | [HTTP service layer](stories/language-runtime-database/10-http-service.md) | ⬜ | Hold |
|
||||
| 11 | [Fibers](stories/language-runtime-database/11-fibers.md) | ⬜ | Hold |
|
||||
| 12 | [Blue-green deploy](stories/language-runtime-database/12-blue-green-deploy.md) | ⬜ | Hold |
|
||||
|
|
@ -332,6 +333,7 @@ Ecommerce sample (verified 2026-06-13): `api.rest` 17/17 expected statuses pass.
|
|||
| 9d | Keypair attach auth — mutual challenge–response, grants name public keys, uid superseded | **no spec yet** — four forks recorded; plan folds into 9c's |
|
||||
| 9e | Durability + throughput + scale — restart-persistence, read/write benchmark, ~1M rows; the gate every later optimization re-runs | **no spec yet** — four forks recorded; the measurement backbone |
|
||||
| 9f | io_uring group-commit write path — batched durability overlapped on shard threads, fsync fallback | **no spec yet** — brainstorm after iterations 8 + 9e |
|
||||
| 9g | Query grammar from real embedded-DB corpora — whole-query count + correlated exists, driven by the skillhost SQL catalogue; add only what a corpus uses | **no spec yet** — three forks; may collapse to "confirm len(query) + add exists" |
|
||||
| 10 | HTTP service layer | [plan 6](superpowers/plans/2026-08-01-http-service-layer.md) |
|
||||
| 11 | Fibers | vision §3, [blue-green exploration](plan/exploration/blue-green-vm/00-vision.md) |
|
||||
| 12 | Blue-green deploy | [spec](superpowers/specs/2026-08-03-blue-green-vm-design.md) — plan authored after iterations 9–10 |
|
||||
|
|
|
|||
|
|
@ -57,6 +57,7 @@ iterations); no commits by agents — drafts go to `.dev/commit.md`.
|
|||
| 9d | [Keypair attach auth](09d-keypair-attach-auth.md) | program identity is a keypair: mutual challenge–response at attach, grants name public keys, replay-proof, rotation is a config change |
|
||||
| 9e | [Durability, throughput, scale](09e-durability-throughput-scale.md) | restart-persistence proof, read/write benchmark, ~1M-row load; the measurement gate every optimization signs |
|
||||
| 9f | [io_uring group-commit](09f-io-uring-commit.md) | replace fsync-per-commit with io_uring batched durability, overlapped on the shard threads; fsync fallback kept |
|
||||
| 9g | [Query grammar corpus](09g-query-grammar-corpus.md) | grow the query grammar from real embedded-DB apps: `count`/existence subqueries from the skillhost corpus; add only what a corpus uses |
|
||||
| 10 | [HTTP service layer](10-http-service.md) | `service` blocks route to VM methods; REST parity with Stage 2 |
|
||||
| 11 | [Fibers](11-fibers.md) | green threads on the shard scheduler: reduction-budget preemption, park on I/O |
|
||||
| 12 | [Blue-green deploy](12-blue-green-deploy.md) | two VM slots, in-runtime compile, atomic switch, resident rollback |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,161 @@
|
|||
# Iteration 9g — query grammar, driven by real embedded-DB corpora
|
||||
|
||||
> Format: `product/story-iteration-template`. Part of
|
||||
> [Story — one language, one runtime, one database, one binary](00-story.md).
|
||||
>
|
||||
> **Inserted 2026-08-16.** A query-surface iteration in the 9b family: the
|
||||
> language-integrated query grows to cover the grammar that *real
|
||||
> applications backed by an embedded SQL database actually use* — measured,
|
||||
> not guessed, by cataloguing a real app and adding only the constructs it
|
||||
> depends on. The method is the postgres/System.Linq reference pattern
|
||||
> applied to a whole application: **an embedded-SQLite app is a grammar
|
||||
> corpus; each one analysed drives a grammar increment.**
|
||||
>
|
||||
> **No spec exists yet.** The forks in *Info* are genuine decisions.
|
||||
|
||||
## Why this iteration exists
|
||||
|
||||
The 9b query surface ships scan / `where` / `select` / `order by` / `take`,
|
||||
`ref`/`backlink` navigation, insert / update / delete, `@unique`, and FK
|
||||
restrict (all running, `docs/examples/employee`). What it does NOT yet cover
|
||||
is everything past that — and "everything" is unbounded, so the sensible way
|
||||
to choose the *next* grammar is to point at a real program that uses an
|
||||
embedded SQL database and add exactly what it needs.
|
||||
|
||||
**Corpus #1: `~/projects/skillhost`** (a C++ MCP host, embedded SQLite as its
|
||||
in-memory skill catalog; surveyed 2026-08-16). Its entire SQL footprint is
|
||||
one file (`src/catalog/catalog.cpp`, 172 lines): one table + index, a
|
||||
single-row parameterized `INSERT`, and four `SELECT`s. Mapping each statement
|
||||
to the writeonce query surface:
|
||||
|
||||
| skillhost statement | writeonce today |
|
||||
| --- | --- |
|
||||
| `INSERT INTO skills (…) VALUES (?,…)` | ✅ `insert Skill { … }` |
|
||||
| `SELECT … WHERE name = ?` | ✅ `from s in Skill where s.name == n select s` (unique-index probe) |
|
||||
| `SELECT … ORDER BY name` | ✅ `from s in Skill order by s.name select s` |
|
||||
| `SELECT COUNT(*) FROM skills` | ❌ **whole-query `count`** |
|
||||
| `SELECT … WHERE NOT EXISTS (SELECT 1 FROM skills c WHERE c.parent = s.name) ORDER BY name` | ❌ **correlated `not exists` subquery** |
|
||||
|
||||
So the real grammar gap this corpus demands is **two constructs**, and —
|
||||
importantly — *neither is the parked full group-by/projection machinery*.
|
||||
Everything else SQLite offers (JOIN, HAVING, LIMIT/OFFSET, DISTINCT, CTE,
|
||||
window functions, UNION, UPSERT, RETURNING, JSON1, FTS5, triggers, generated
|
||||
columns) skillhost does not touch, so none of it is in scope here.
|
||||
|
||||
## Goals
|
||||
|
||||
- **Whole-query `count`**: `count(from s in Table [where …] select …)` yields
|
||||
an `Int` — the trivial, group-free special case of aggregation (materialize
|
||||
the query, take its length). It is a stepping stone toward, and independent
|
||||
of, the parked group-by aggregation.
|
||||
- **Existence subqueries**: `exists(<query>)` and `not exists(<query>)` as a
|
||||
boolean, usable in a `where` guard, where the inner query may reference the
|
||||
outer range variable (a *correlated* subquery — skillhost's roots-of-the-
|
||||
tree query). Short-circuits: existence needs only the first matching row.
|
||||
- **Parity, proven by translation**: a new `docs/examples/skill-catalog`
|
||||
sample mirrors skillhost's schema and expresses all five of its statements
|
||||
in writeonce, producing results identical to what skillhost's SQLite
|
||||
returns for the same data.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- What to achieve?
|
||||
- **Given** `count(from s in Skill select s)` and
|
||||
`count(from s in Skill where s.parent == nil select s)`,
|
||||
- **when** compiled and run,
|
||||
- **then** each yields the correct row count as an `Int`, lowered to a
|
||||
materialize-then-length over the existing scan/where loop — no group
|
||||
machinery, provable by disassembly.
|
||||
- What to achieve?
|
||||
- **Given** `from s in Skill where not exists(from c in Skill where
|
||||
c.parent == s.name select c) order by s.name select s` — the roots of
|
||||
the skill tree,
|
||||
- **when** run over a catalog with parent/child skills,
|
||||
- **then** it returns exactly the childless skills in name order, and the
|
||||
inner query correctly sees the outer `s` (correlation), matching
|
||||
skillhost's `NOT EXISTS` result row-for-row.
|
||||
- What to achieve?
|
||||
- **Given** the `skill-catalog` sample seeded with the same rows a
|
||||
skillhost session would load,
|
||||
- **when** each of skillhost's five catalog operations is run through the
|
||||
writeonce translation,
|
||||
- **then** every result matches, and the sample's README records the one
|
||||
translation choice made (see fork 1).
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- **Full group-by aggregation** (`group … by … into g … select { count(g),
|
||||
avg(g.f) }`) — still parked (9b's deferral). Whole-query `count` here is the
|
||||
degenerate case, not the general one; `sum`/`avg`/`min`/`max` as query
|
||||
aggregates ride with the group-by iteration.
|
||||
- **Every SQL construct skillhost does not use**: JOIN, HAVING, LIMIT/OFFSET
|
||||
(writeonce has `take`; `skip`/offset waits for a workload), DISTINCT, CTE /
|
||||
`WITH RECURSIVE`, window functions, UNION/INTERSECT/EXCEPT, UPSERT /
|
||||
`ON CONFLICT`, `RETURNING`, multi-row `VALUES`, `INSERT … SELECT`, JSON1
|
||||
operators, FTS5, triggers, generated columns, explicit collation. Each
|
||||
enters only when a corpus demands it — that is this iteration's whole
|
||||
method.
|
||||
- **A resident SQL parser** — the doctrine stands: skillhost is a grammar
|
||||
*corpus* to translate against, never a syntax writeonce adopts. No SQL text
|
||||
in a compiled image.
|
||||
- **Subqueries in general** beyond correlated `exists`/`not exists` (e.g. a
|
||||
subquery producing a value, `IN (subquery)`, scalar subqueries) — add when
|
||||
a corpus uses them.
|
||||
|
||||
## Info
|
||||
|
||||
Forks the spec must settle:
|
||||
|
||||
**1. Does skillhost's `NOT EXISTS` even need a subquery, or does a `backlink`
|
||||
express it?** skillhost's `skills` table is self-referential (`parent` → a
|
||||
`name`), and its roots query is "skills no other skill names as parent." In
|
||||
writeonce that is naturally a **backlink emptiness**: give `Skill` a
|
||||
`children: backlink Skill.parent` and write `where len(s.children) == 0` — no
|
||||
subquery at all, using machinery that already exists (backlink probe) plus a
|
||||
`len` on the result. So the corpus may be fully expressible *today* once
|
||||
`count`/`len` over a query lands, making `exists` strictly optional for
|
||||
skillhost. Leaning: ship whole-query `count`/`len` (needed regardless), and
|
||||
add `exists`/`not exists` as the general construct for correlations a backlink
|
||||
cannot express (a correlation on a non-relation column) — but let the
|
||||
`skill-catalog` sample use the idiomatic backlink form for its roots query and
|
||||
record the subquery form as the alternative. This keeps the new surface
|
||||
minimal and honest about what the corpus actually forces.
|
||||
|
||||
**2. `count` vs `len`.** writeonce already has `len`/`count` builtins on a
|
||||
`multi`. A query yields a `multi`, so `len(from … select …)` may already work
|
||||
with no new surface at all — the "gap" could be purely that a bare query in
|
||||
argument position typechecks and lowers. Leaning: verify `len(<query>)` works
|
||||
end to end first; if it does, whole-query `count` is a documentation/alias
|
||||
matter, not new code, and the only real new construct in this iteration is the
|
||||
existence subquery (fork 1's optional half). The spec must confirm this
|
||||
against the running compiler before committing scope.
|
||||
|
||||
**3. Correlated-subquery execution.** If `exists` lands, the inner query
|
||||
references the outer row, so it re-evaluates per outer row (a nested loop) or
|
||||
uses the referenced index. Leaning: nested-loop for correctness first (the
|
||||
data is small; skillhost's catalog is dozens of skills), index-backed probe as
|
||||
the optimization the spec records — mirroring how 9b did scans before index
|
||||
selection.
|
||||
|
||||
**Method note (the durable part):** this iteration establishes the pattern for
|
||||
*all* future query-grammar growth — catalogue a real embedded-DB application,
|
||||
add only the constructs it uses, translate its statements 1:1 as the
|
||||
acceptance, and park the rest by name. skillhost is corpus #1 and, tellingly,
|
||||
needs almost nothing beyond what 9b already shipped — which is itself the
|
||||
strongest evidence that the 9b surface was scoped right.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
- **Brainstorm the spec** settling the three forks — especially forks 1/2,
|
||||
which may collapse the iteration to "confirm `len(<query>)` works + add
|
||||
`exists`," a very small increment.
|
||||
- **Acceptance workload**: `docs/examples/skill-catalog` — `Skill { name:
|
||||
Text @unique, description: Text, location: Text, root: Text, parent: ?ref
|
||||
Skill, children: backlink Skill.parent }` and a CLI mirroring skillhost's
|
||||
catalog operations (add, get-by-name, list, list-roots, count), each a
|
||||
direct translation of the corresponding SQLite statement, with an
|
||||
acceptance script asserting the same results skillhost produces.
|
||||
- Expected shape: small parser/type/emit additions for `exists`/`not exists`
|
||||
(a query in boolean position, correlated), whole-query `count`/`len`
|
||||
confirmed or wired, and the sample + script. No engine format change beyond
|
||||
what 9b already appended.
|
||||
Loading…
Reference in a new issue