writeonce/docs/plan/oop-vm/04-db-binding.md

# DB binding — row format, id discipline, WAL layout, query subset

> Normative companion to the engine plan
> ([`2026-08-01-db-engine-binding.md`](../../superpowers/plans/2026-08-01-db-engine-binding.md)),
> the way `00-wob-format.md` is normative for the image. Grows with the
> plan's tasks; this revision covers **Task 1 (row storage)**. Memory-safety
> doctrine lives in the 9b design's section 6 (the copy bulkhead) — this doc
> is the *format*.

## Two memory worlds, one crossing rule

Rows store **no VM pointer**, ever. Values cross from VM heap to row storage
by copy on insert, and back by copy on read (`wo_row_read` allocates fresh VM
values from the shard's runtime). The engine's own allocations are plain
malloc — never the VM arena, so table growth cannot eat the program's heap
cap, and a heap-exhausted program can still read its data.

## Row format

```
row      := header slots
header   := id u64 | class_id u32 | flags u32          (16 bytes)
slots    := field_cnt × u64, declaration order          (the VM object shape)
```

One 8-byte slot per field, kind-driven — the same kind bytes the `.wob`
class table carries, walked the same way the VM walks them:

| kind | slot holds | engine-owned shape |
| --- | --- | --- |
| `SCALAR` | the 8 bytes themselves | — (`WO_NIL_SCALAR` spells a `?scalar` nil) |
| `TEXT` | pointer, 0 = nil | `db_text { len u32; bytes[] }` |
| `OWNED` | pointer, 0 = nil | `db_rec { class_id u32; slots[] }` — flattened by value, recursively through these same rules |
| `MULTI` | pointer, 0 = nil | `db_multi { elem_kind u8; len u32; items[] }`, elements encoded element-wise |
| `MAP` | pointer, 0 = nil | `db_map { key_kind, val_kind u8; len u32; kv pairs }` |
| `GCREF` | **never stored** | compile error upstream (the GC bulkhead); the engine refuses it defensively as an encode error |

`ref T` is a `SCALAR` at this layer — the target row's id. The engine learns
what it references only when the FK checks land (9b plan, Task 3).

## Storage

Per shard, per class, created lazily on first insert:

- **Slabs** of 256 rows (`DB_SLAB_ROWS`), malloc'd, **never moved or freed
  while the table lives** — a row's address is stable for its lifetime,
  which is the property 9b's loop-scoped row views stand on.
- An **occupancy bitmap** (one bit per slot, slab-major) and a LIFO
  **free-slot list**: removal recycles the slot; a recycled slot is always
  used before a new slab grows. Ids are never reused; slots are.
- The **primary index**: an open-addressing hash, id → slot, splitmix64
  finalizer, power-of-two capacity, 0.7 load, tombstoned deletes (ids are
  never 0 and never reused, so the all-ones sentinel cannot collide).

## Id discipline

Per table, per shard: shard S of N allocates `S+1, S+1+N, S+1+2N, …` — the
c-runtime plan's shipped interleave. Creation is coordination-free; a row's
owner shard is `(id-1) % N`. Milestone 1 runs at N=1 and everything
degenerates to `1, 2, 3, …`. Id 0 does not exist (it is the hash's "empty"
and the `?ref`'s nil).

## Choke points

`wo_row_insert` and `wo_row_remove` are the only functions that mutate a
table. Task 4's secondary indexes hook exactly these two sites (marked
`INDEX HOOK` in `database/src/table.c`); the WAL (Task 2) stages its record
beside the same calls. Anything else touching a slab is a defect by
definition — the doctrine the Rust engine learned and this engine enforces.

## WAL (Task 2) — `database/src/wal.{c,h}`

Record framing, replay-whole-or-not-at-all:

```
record  := len u32 | crc u32 | payload | mark u32
len      = payload bytes (never 0: a zero length is the preallocated tail)
crc      = CRC32 (poly 0xEDB88320) of the payload
mark     = 0x574F4C31 "WOL1", the last bytes of the record — a record
           without its mark is torn by definition
payload := kind u8 | class_id u32 | row_id u64 | body
kind     = 1 insert (body = fields), 2 remove (no body), 3 update (Task 5)
```

Body fields walk the class table's kinds: `SCALAR` 8 bytes; `TEXT` u32 len +
bytes (`0xFFFFFFFF` = nil); `OWNED` presence u8 then class id + fields
recursively; `MULTI` presence + elem kind + len + elements; `MAP` presence +
both kinds + len + pairs. Little-endian, same platform note as the loader.

**Commit order (doctrine, verbatim from the shipped phase-D pattern):** RAM
apply → stage record → `wo_wal_commit` (one pwrite of the batch + one
fdatasync) → only then acknowledge. Group commit = everything staged since
the last commit rides one sync.

**Replay** decodes payloads straight into engine-owned values — no VM heap
involved, boot cannot depend on a VM existing — and rows re-enter through
the choke-point row API, so Task 4's indexes rebuild for free. A torn tail
(short record, bad CRC, missing mark, zero length) ends the intact prefix:
everything from the tear on is dropped whole, and `wo_wal_open` positions
its write offset AT the tear so the next commit overwrites it. A record that
CRC-passes but does not decode is corruption, not a tear — replay fails
loudly. A missing file is a fresh boot, not an error. After replay each
table's `next_id` sits past every replayed id this shard owns.

**Oracle:** `wo_wal_check(path)` walks a file with no engine and reports the
intact record count and prefix end — the crash battery's verifier
(`runtime/test/test_wal.c`: five rounds of insert/commit/ack-over-pipe with
SIGKILL mid-stream; every acked row present and exact after replay).

**Where the log lives (databasev2 7, 2026-09-10).** `WO_DATA` is always a
path, never a sentinel (ephemerality is `WO_EPHEMERAL=1`, databasev2 2 task
6a), and it names the store in one of two forms, resolved by
`wo_wal_resolve_data_path` (`wal.{c,h}`) before anything is opened:

- **Directory form** — an existing directory, or any path ending in `/`: the
  log is `<dir>/shard-0.wal`, byte for byte what every deployment and gate
  before this iteration used (a trailing slash still yields the `//` the
  pre-7 driver produced, and a trailing slash on a missing directory still
  fails at open: `wovm: cannot open <dir>//shard-0.wal`, exit 2).
- **File form** — anything else: the path IS the log (`app.db`, `store.wo.db`
  — the name is the operator's). An existing regular file is opened; an
  absent path is created by `wo_wal_open`, but only when its parent directory
  already exists. Two refusals, each exit 2 and ONE stderr line: a parent
  that is not an existing directory — `wovm: WO_DATA=<path> — its parent
  <parent> is not an existing directory; create it first (wovm never runs
  mkdir -p).` — because a typo must not plant a store somewhere unexpected;
  and a path that exists but is neither a regular file nor a directory
  (fifo, socket, device). A result longer than the driver's path buffer is
  refused as well, never truncated.

One file is the whole store at any core count (shard 0 is the only WAL
writer since arc stage 3). Compaction (databasev2 3) and schema migration
(databasev2 12) rewrite through `<log path>.compact` beside the log and
fsync the log's parent after the `rename` — both derive that from the log
path, never from `WO_DATA`, so the file form inherits their crash safety
unchanged; the boot-time parent check and the post-rename fsync share one
derivation (`parent_dir_of`). `WO_EPHEMERAL` set together with either form
refuses exactly as 6a says. Pinned by `runtime/test/test_wal.c`
`test_resolve_data_path` and `test_file_form_temps_beside_log`.

## Insert (Task 3) — builtin 61, `database/src/db.c`

`insert Class { field: expr, … }` is a typed expression (statement position
included): fields validate like a constructor literal (defaults and `?`
fields omittable — an omitted `?scalar` gets `WO_NIL_SCALAR`, other omitted
optionals the zero word, declared defaults their value), and the result is
the new row's id. Lowering emits builtin **61**: R[B] = class-id constant,
R[B+1..] = one slot per declared field in declaration order (the literal's
order is irrelevant — slots are the class table's).

Execution: `wo_row_insert` (RAM, engine copies every value), then — when
durability is on — stage, then a barrier before the acknowledgment. **Updated
2026-08-28 (databasev2 4 part A): group commit landed, and the barrier's
location now depends on which path the statement takes.**

A statement arriving from a worker shard marshals to shard 0 and parks; shard 0
stages every such request, issues **one** barrier when its queue empties, and
only then releases the held replies — so each writer is acknowledged after the
barrier that carried *its* record. A statement already running on shard 0 takes
the inline path and still commits before the builtin returns, because it has no
reply to hold: it returns into its own fiber, and batching it would require
parking that fiber on the barrier (deferred to part B). The boundary is the
queue draining, **not** the tick this document previously anticipated — a tick
would add latency to a lone writer, taxing an idle system to serve a busy one.

Measured: ~2.9× durable write throughput and ~2.1× lower p50 on a
write-concurrent workload; unchanged for a serial writer, which has nothing to
batch with.

**Compaction (databasev2 3, 2026-08-29) may run only where NOTHING IS STAGED.**
That is a correctness requirement, not a scheduling preference: the staging
buffer holds records destined for a file that compaction is about to replace, so
compacting with a non-empty buffer would either write them into a file about to
be discarded or lose them with it. In practice the safe points are immediately
after a barrier — the drain's, and the inline path's — and both are wired.
`wo_wal_compact` refuses a non-empty buffer as a backstop rather than trusting
its callers.

**Recovery is unchanged by compaction.** The result is an ordinary log in the
ordinary record grammar, replayed from byte 0; there is no snapshot, no second
source, no cutoff offset and no control file. Crash safety comes from `rename`
being atomic: before it the live log is intact and the temp file is not
authoritative, after it the new log is complete, and no reader can observe a
mixture. A crash mid-rewrite leaves a temp file, which the next open removes.

A failed compaction is a **missed optimisation, not a durability event** — the
original log is left usable and the process continues. It must not take the
fatal path below.

A failed commit **no longer traps — it ends the process** (exit 74, with a
diagnostic naming the operation, log path, `errno` and batch size). So does a
failed staging. `WO_T_IO` is unreachable from a DB write. One rule: once a
statement has mutated RAM, the outcomes are durable or death. Engine failures
still trap `WO_T_DB`. Durability is opt-in: `WO_DATA=<dir>` makes the CLI replay
`<dir>/shard-0.wal` before the entry runs and commit every insert; without
it the engine is RAM-only (every corpus fixture runs that way).

Ownership: the engine copies at the row API, so an insert **borrows** its
field values — no transfer, no E304; freshly built values are dropped at the
site (emit.ml mirrors the push/set reap). The insert node is trap-capable
(unique violations arrive with Task 4) and carries a live-mask drop entry.

## Still to come in this document

- **Task 4**: secondary-index format, `@unique` trap code.
- **Task 5**: the select subset, update record semantics, and its builtins.