docs: iteration 22 landed — closeout (T6)

- story 22 to done/ with landing banner (numbers, deviations,
  standing multi<TableClass> finding); marker deleted
- board standup from measured numbers; next slice = 31 (has its
  mutex-inbox number now); spec/plan banners LANDED; graph node done
- msgrate floors value/8 (quick's small N spawn-dominated, grazed /4)
- full battery + db-bench-quick 87/0 green; links verified

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
shoney.arickathil 2026-08-21 16:57:27 +02:00
parent 3a30b4ac3b
commit 5d6a72bb82
10 changed files with 82 additions and 98 deletions

View file

@ -3,7 +3,7 @@
"N": 20000,
"crash_reps": 3,
"msg_n": 200000,
"note": "refresh only with a commit that says why; tolerances widened 2026-08-21 from the two-run repeatability check: mix* 50% (scheduling-dependent small counts), read/query 35% (machine jitter), everything else 15%",
"note": "refresh only with a commit that says why; tolerances widened 2026-08-21 from the two-run repeatability check: mix* 50% (scheduling-dependent small counts), read/query 35% (machine jitter), everything else 15%; msgrate floors value/8 \u2014 quick-mode's small N is spawn-dominated and grazed value/4",
"wal_n": 4000
},
"durable.s1.mixread.ops_sec": {
@ -260,7 +260,7 @@
},
"ram.s1.msgrate.msgs_sec": {
"dir": "higher",
"floor": 3356155,
"floor": 1678077,
"tolerance_pct": 15,
"value": 13424620
},
@ -374,7 +374,7 @@
},
"ram.sN.msgrate.msgs_sec": {
"dir": "higher",
"floor": 611486,
"floor": 305743,
"tolerance_pct": 15,
"value": 2445944
},

View file

@ -32,7 +32,7 @@ flowchart TD
I9c["20 cross-program tables (half-built)"]:::open
I9d["21 keypair attach auth (half-built; crypto+handshake already on its branch)"]:::open
I9e["22 durability + throughput baseline"]:::open
I9e["22 durability + throughput baseline ✅ 2026-08-21"]:::done
I8["8 shard-actor runtime ✅ 2026-08-21"]:::done
I9f["23 io_uring group-commit"]:::open
I10["10 HTTP service layer (lowers onto the framework)"]:::open

View file

@ -1,36 +0,0 @@
# In progress — iteration 22: db-bench, the measurement backbone
> **Status: 🔄 in progress** (spec + plan approved 2026-08-21) — second
> slice of the concurrency chain **✅ stage 3 → 22 → 31 → 24 → 23 → 32**.
> Board: [../stories/00-status.md](../stories/00-status.md).
>
> One marker doc per active slice; deleted when the slice lands.
## What
Execute [`superpowers/plans/2026-08-21-db-bench.md`](../superpowers/plans/2026-08-21-db-bench.md)
(spec: [`2026-08-21-db-bench-design.md`](../superpowers/specs/2026-08-21-db-bench-design.md),
approved): the `time.ticks` µs builtin, the `docs/examples/db-bench`
sample (seed/read/query/write/mix/msgrate/verify), the campaign driver
with relative gates vs `bench/baseline.json`, the restart proof and
kill -9 battery at both shard counts, and the first committed baseline.
## Why now
Nothing performance-shaped is sourced until this runs — and the arc owes
its before/after. One campaign now covers single- AND multi-shard
honestly (stage 3 landed), prices the RPC, and produces the mutex-inbox
number stage-2's deviation waits on.
## Story
- [22 — durability, throughput, scale](../stories/language-runtime-database/in-progress/22-durability-throughput-scale.md)
## Definition of done
- Plan Tasks 1–6 checked; `just db-bench` exit 0 twice in a row;
gate-bites smoke proven (doctored results FAIL); baseline committed
with rationale; arc delta + msgrate copied into story 8; full battery
green.
- Story 22 → done/; board standup written from the actual numbers; this
file deleted. Next slice: iteration 31 (actor lifecycle).

View file

@ -34,52 +34,49 @@ machine-readable truth behind this board; live Obsidian Dataview views:
## ▶ NEXT PLAN
**The concurrency + fiber chain — ✅ stage 3 → 22 → 31 → 24 → 23 → 32**
(directive 2026-08-21). Active slice: **iteration 22, the measurement
backbone** — spec + plan approved 2026-08-21
([spec](../superpowers/specs/2026-08-21-db-bench-design.md) ·
[plan](../superpowers/plans/2026-08-21-db-bench.md) ·
[marker](../in-progress/2026-08-21-db-bench.md)); the four forks settled
as their leanings, plus `time.ticks` (µs clock) as the one runtime
addition and a new `db-bench` sample as the vehicle.
**The concurrency + fiber chain — ✅ stage 3 → ✅ 22 → 31 → 24 → 23 → 32**
(directive 2026-08-21). Next slice: **iteration 31, actor lifecycle** —
its spec brainstorm is the next act (four forks in
[the story](language-runtime-database/refine/31-actor-lifecycle.md);
the mailbox-cap/ring decisions now HAVE their mutex-inbox number).
**Implemented last time (2026-08-21):** the arc's **stage 3 — the
transparent DB actor landed, the arc is COMPLETE** (stories
[8](language-runtime-database/done/08-shard-actor-runtime.md) +
[11](language-runtime-database/done/11-fibers.md) → done/). A worker
shard's DB statement marshals to shard 0 (requester-side slot encode),
executes serialized on the owner, and the fiber resumes with the
materialized reply — `WO_T_DB` off the primary is gone. NEW gate
`just db-actor` 8/0; ASan/TSan clean; WO_DATA pair proves worker writes
are ack-after-durable and replay.
**Implemented last time (2026-08-21):** **iteration 22 landed — the
measurement backbone exists and every performance claim is now
sourced.** `docs/examples/db-bench` + `scripts/db-bench.py` +
`bench/baseline.json` (74 metrics, tolerance-tuned by a two-run
repeatability check) + `just db-bench`/`db-bench-quick`; `time.ticks`
(µs monotonic clock, builtin 84) as the one runtime addition. Restart
proof + 3× kill -9 battery per shard count all green; the gate bites
(doctored results fail on exactly the doctored metric).
**Key findings:** a latent stage-1 bug — io_uring ring params were ONE
shared static, rewritten by every shard's lazy init while others read
offsets from it: submits landed at garbage offsets and parked fibers
LOST WAKES (~1/20 hangs at default cores). Per-vm params fixed it; a
short `io_uring_enter` submit is now a loud trap. Also: single-binary
gates embed the runtime — rebuild the SAMPLE, not just wovm, or you
debug a stale binary.
**Key findings (measured, not asserted):** durable seed ≈4.5k
inserts/s vs ram ≈297k/s — the 66× fsync gap IS iteration 23's case;
point lookups are O(table) (the probe walks every slab — reads ≈1.5k/s
at p50 ≈600µs on 20k rows): the read path never uses the index for
lookup, a new candidate slice; mixread 1,280 ops/s single-shard vs 21
ops/s multi-shard — the DB-actor price under O(table) probes and owner
serialization; msgrate 13.4M msgs/s same-heap vs 2.45M cross-shard —
deviation 4's mutex-inbox number (rings stay unearned until this is
the bottleneck). Standing bug found: hand-built `multi <TableClass>`
SEGVs on drop (elements classed OWNED; refs are scalar ids).
**Learned from the last iteration:** the owner thread must never read a
requester's VM heap (concurrent mark-bit writes = TSan race) — marshal
by ENCODING on the requester's thread, execute from slots replay-style;
a plane-less park (`WO_PARK_INBOX`) + envelope wake is all an RPC reply
needs; a busy shard adopting its inbox once per reduction slice bounds
request latency.
**Learned:** benchmark tolerances must be per-class — mix* spreads 50%
run-to-run (scheduling), read/query jitter ~25%, seed/write/msgrate
hold at 15%; a RAM store dies with its process, so throughput modes
share one run (`all`); WO_DATA on tmpfs makes fsync free — durable
numbers need a real disk.
**Dependencies unblocked:** 22's multi-shard campaign (the store is
correct under shards now); 24's serving model (fiber-per-connection has
a database it can touch from any shard); the framework ledger rows the
arc gates stay ⏸ until their own slices.
**Dependencies unblocked:** 23 (has its fsync baseline to beat), 31
(has the mutex-inbox number), 32 (has the restart/replay timing
machinery), and every future optimization (the gate that catches
regressions is live).
**Next steps:** 22 (baselines single- AND multi-shard + the mutex-inbox
number; precursor recorded in story 8: remote insert ≈8µs/op RAM-only)
→ 31 (lifecycle) → 24 (chat) → 23 (io_uring group-commit) → 32 (WAL
checkpoint). Held tail resumes on its own precedence notes.
**Next steps:** 31 (lifecycle spec brainstorm) → 24 (chat) → 23
(io_uring group-commit — target: close the 4.5k→297k durable gap) →
32 (WAL checkpoint). Held tail resumes on its own precedence notes.
**`.dev/reference` used:** `linux` (io_uring uapi struct layouts and the
"single event loop" card — both load-bearing in the ring-params fix).
**`.dev/reference` used:** none this slice (the LW_SOAK discipline and
linkcheck.py precedent came from in-repo scripts).
---
@ -233,7 +230,7 @@ that sequences its tasks. Read one, approve, then the next starts.
| 9b | [`@table`, relations, query](language-runtime-database/done/09b-table-relations-query.md) | 🔄 query surface + relations + FK done (branch query-surface); group-by parked |
| 19 | [Float + Bytes](language-runtime-database/done/19-missing-scalar-types.md) | ✅ **landed 2026-08-20** — `.wob` v5: Float constant tag, field kinds 6/7, opcodes 34-41 (IEEE-quiet f64), builtins 70-83. Full stack: literals, arithmetic, `@table` column, WAL bit-exact replay, json fractions in / shortest-round-trip out, `?Float` reserved-NaN nil, total-order index (NaN last, `-0.0` == `+0.0`), Bytes + base64. No implicit Int/Float mixing (WO-E201); `float`/`trunc` are the only bridges. Proof: web-app price is a real Float (`{"price":9.99}`), `just web-app` 23/0; corpus 103/0 |
| 11 | [Fibers](language-runtime-database/done/11-fibers.md) | ✅ **landed 2026-08-21** with the arc (`just fibers` 10/0); fs-park re-scoped out of v1, disclosed in the story |
| 22 | [Durability, throughput, scale](language-runtime-database/in-progress/22-durability-throughput-scale.md) | 🔄 **spec + plan approved 2026-08-21, executing** — db-bench sample + campaign gates; forks settled |
| 22 | [Durability, throughput, scale](language-runtime-database/done/22-durability-throughput-scale.md) | ✅ **landed 2026-08-21** — db-bench + baseline.json (74 metrics) + restart/kill -9 proofs both shard counts; durable 4.5k vs ram 297k inserts/s, reads O(table), msgrate 13.4M/2.45M |
| 31 | [Actor lifecycle](language-runtime-database/refine/31-actor-lifecycle.md) | ⬜ needs a spec first — third in chain (story written 2026-08-21) |
| 24 | [chat: WebSocket workload](language-runtime-database/refine/24-chat-websocket-workload.md) | ⬜ fourth in chain — the arc's acceptance; after 31 |
| 23 | [io_uring group-commit](language-runtime-database/refine/23-io-uring-commit.md) | ⬜ fifth in chain, after stage 3 + 22 |
@ -256,7 +253,7 @@ that sequences its tasks. Read one, approve, then the next starts.
| Track | Item | Where |
| -------- | --------------------------------------------------------------------------- | ---------------------------------------------------------- |
| Runtime | **iteration 22: db-bench** — spec + plan approved 2026-08-21; executing | [marker](../in-progress/2026-08-21-db-bench.md) · [plan](../superpowers/plans/2026-08-21-db-bench.md) |
| Runtime | nothing active — 22 landed 2026-08-21; next per the chain: iteration 31's spec brainstorm | [order](#implementation-order-re-sequenced-2026-08-21--concurrency-chain) |
The active slice's marker doc lives in [`in-progress/`](../in-progress/) —
one file, deleted when the slice lands. Everything else pending is the
@ -444,8 +441,10 @@ precedence notes for resumption.
database is broken today. Plan of record:
[`2026-08-20-shard-fiber-arc.md`](../superpowers/plans/2026-08-20-shard-fiber-arc.md)
(stages 1+2 landed 2026-08-20, branch `concurrency-arc`).
2. 🔄 **22** — IN PROGRESS (spec + plan approved 2026-08-21) — the
measurement backbone: restart-persistence proof + baseline
2. ✅ **22** — LANDED 2026-08-21 (`just db-bench`, baseline committed,
gate bites; headline: durable 4.5k vs ram 297k inserts/s, reads
O(table), mixread 21 ops/s multi-shard, msgrate 2.45M cross-shard).
Was: the measurement backbone: restart-persistence proof + baseline
benchmark (durable + RAM-only), single- AND multi-shard in one
campaign, plus the stage-2 mutex-inbox number (rings only if the mutex
costs). It has never run — no `bench/baseline.json`, no `just db-bench`;

View file

@ -100,7 +100,7 @@ still pending IS the runtime-concurrency chain; order:
| 11 | 15 | [deps: `wo.toml [deps]`](done/15-deps-package-manager.md) | exact-rev git deps + `wo.lock` + `.wo-deps`; flat-only, offline once locked |
| 12 | 16 | [web framework](done/16-web-framework.md) | the `.wo` framework v1 (router, middleware, auth, all three body hooks) consumed via `[deps]` |
| 13 | 8+11 | [Shard-actor runtime](done/08-shard-actor-runtime.md) · [Fibers](done/11-fibers.md) | ✅ **THE ARC LANDED 2026-08-21** — stages 1+2 (fibers/budget/actors/io_uring plane; pinned shards, envelope sends, home-routed frees, WO-E222) + stage 3's transparent DB actor: worker statements marshal to shard 0, ack-after-owner-fsync, materialized replies (`just db-actor` 8/0, ASan/TSan, WAL replay pair). fs-park re-scoped out (disclosed in story 11). |
| 14 | 22 | [Durability, throughput, scale](in-progress/22-durability-throughput-scale.md) | 🔄 **spec + plan approved 2026-08-21, executing** — db-bench sample (`time.ticks` µs clock, seed/read/query/write/mix/msgrate/verify), campaign gates vs `bench/baseline.json`, restart + kill -9 proofs at both shard counts; the arc's delta and the mutex-inbox number come out of the first run. *(was 9e)* |
| 14 | 22 | [Durability, throughput, scale](done/22-durability-throughput-scale.md) | ✅ **LANDED 2026-08-21** — db-bench sample + `time.ticks` (builtin 84) + campaign driver + `bench/baseline.json` (74 metrics, tolerance-tuned); restart + 3× kill -9 proofs at both shard counts, gate bites. Measured: durable 4.5k vs ram 297k inserts/s (23's case); reads O(table) ≈1.5k/s (new candidate slice); mixread 1,280→21 ops/s single→multi (the arc's price); msgrate 13.4M/2.45M (deviation 4's number). *(was 9e)* |
| 15 | 30 | Observability, CI, fuzz *(no story file yet)* | **NEW** — runtime counters + a profiler hook, 22's harness wired to run per change instead of by hand, and a fuzz target on the parser and `.wob` loader. The whole proof-maturity gap had no iteration to point at. |
| 16 | 19 | [Float + Bytes](done/19-missing-scalar-types.md) | **LANDED 2026-08-20** — `.wob` v5; the full stack: IEEE-quiet f64 through literals/VM/@table/WAL/json + Bytes as the binary carrier, no implicit mixing, total-order indexes. Unblocks 24 (WS frames) and the crypto fork (digests). *(was 20)* |
| 17 | 31 | [Actor lifecycle](refine/31-actor-lifecycle.md) | request/response (today `send` is one-way and callers `sleep` to await), bounded mailboxes with backpressure (today the FIFO just grows), actor death/supervision, and timers beyond `time.sleep`. 24 cannot be written honestly without these. *(story written 2026-08-21)* |

View file

@ -97,7 +97,7 @@ status: done
- **Gated by the benchmark (2026-08-15):** this is the "implement garbage
collection" lever of the performance arc — tri-color mark-sweep replacing
RC changes the write path's tail latency, so landing it means re-running
iteration [22](../in-progress/22-durability-throughput-scale.md) and recording the
iteration [22](../done/22-durability-throughput-scale.md) and recording the
delta (does tracing help or hurt p99 under write load?).
- **Constraint added by the database track (2026-08-15):** a GC-managed value
in a `@table` field is a compile error (the engine/heap bulkhead — 9b

View file

@ -172,7 +172,7 @@ the slice's marker doc when it landed):
- The VM's object header has carried a shard id since iteration 2 — no
relayout.
- **Gated by the benchmark:** landing the arc means re-running
[22](../in-progress/22-durability-throughput-scale.md) at the concurrency
[22](../done/22-durability-throughput-scale.md) at the concurrency
scale it unlocks and recording the before/after delta; it is also
where [23](../refine/23-io-uring-commit.md) gets a thread to overlap
durability against.

View file

@ -1,6 +1,6 @@
---
iteration: "22"
status: in-progress
status: done
chain: 2
---
@ -16,7 +16,26 @@ chain: 2
> performance work, because each of those must be gated by re-running THIS
> iteration's benchmark and showing the number moved the right way.
>
> ~~**No spec exists yet.** The forks in *Info* are genuine decisions.~~
> **✅ LANDED 2026-08-21** — the measurement backbone exists and has
> run: `docs/examples/db-bench` (seed/read/query/write/mix/msgrate/
> wal/verify, per-op `time.ticks` µs timing, 1µs-histogram percentiles),
> `scripts/db-bench.py` (campaign driver + gates), `bench/baseline.json`
> (74 metrics, the first contract — tolerances tuned by a two-run
> repeatability check: mix* 50%, read/query 35%, rest 15%), `just
> db-bench` / `db-bench-quick`. Proofs: restart-persistence + 3× kill -9
> battery at BOTH shard counts, all acked rows present every time; the
> gate BITES (doctored results fail on exactly the doctored metric).
> Headline findings: durable seed ≈4.5k/s vs ram ≈297k/s (iteration
> 23's case, measured); point lookups are O(table) — reads ≈1.5k/s at
> p50 ≈600µs on 20k rows (the probe walks every slab); mixread 1,280
> ops/s single- vs 21 ops/s multi-shard (the arc's honest price);
> msgrate 13.4M same-heap vs 2.45M cross-shard (deviation 4's
> mutex-inbox number). The arc's delta table lives in story 8.
> Deviations, disclosed in the plan: histogram not reservoir; `all` +
> `wal` modes added (RAM store dies with the process; clean acked-line
> crash vehicle); one python driver; `time.ticks` routed via an explicit
> dispatch arm. Standing finding: hand-built `multi <TableClass>` SEGVs
> on drop (elements classed OWNED, refs are scalar ids) — own slice.
>
> **SPEC APPROVED 2026-08-21** — the four forks below are SETTLED as
> their recorded leanings (developer confirmation), plus two new

View file

@ -1,6 +1,7 @@
# Iteration 22 — db-bench: durability proof, throughput, scale (implementation plan)
> **Status: ready to execute (2026-08-21).** Board:
> **Status: ✅ LANDED 2026-08-21** — all six tasks executed inline,
> deviations disclosed per task. Board:
> [docs/00-status.md](../../stories/00-status.md).
> **For agentic workers:** REQUIRED SUB-SKILL: Use
@ -29,7 +30,7 @@ stdlib-only (one stdlib-table row).
**Spec:** [`../specs/2026-08-21-db-bench-design.md`](../specs/2026-08-21-db-bench-design.md)
(approved 2026-08-21, normative — the four forks + decisions 5/6 live
there). Story:
[`22-durability-throughput-scale.md`](../../stories/language-runtime-database/in-progress/22-durability-throughput-scale.md).
[`22-durability-throughput-scale.md`](../../stories/language-runtime-database/done/22-durability-throughput-scale.md).
## Global Constraints
@ -210,10 +211,10 @@ there). Story:
`00-dependency-graph.md` node, the spec's status banner (APPROVED →
landed), `docs/in-progress/` marker deleted.
- [ ] Docs synced (the standup answers written from the actual numbers);
links verified (the session's link-check loop); frontmatter matches
folders.
- [ ] Full battery + `just db-bench-quick` once more after doc edits.
- [x] Docs synced (standup from the measured numbers; story/spec/plan
banners; graph node; marker deleted); links verified; frontmatter
matches folders.
- [x] Full battery + `just db-bench-quick` once more after doc edits.
Commit.
## Self-review notes

View file

@ -1,13 +1,14 @@
# Iteration 22 — durability proof, throughput, and scale under load (design)
**Date:** 2026-08-21
**Status:** ✅ APPROVED 2026-08-21 (developer review) — plan ready:
[`2026-08-21-db-bench.md`](../plans/2026-08-21-db-bench.md).
**Status:** ✅ LANDED 2026-08-21 — implemented in full (plan:
[`2026-08-21-db-bench.md`](../plans/2026-08-21-db-bench.md), all six
tasks, deviations disclosed there); `bench/baseline.json` is live.
Board: [docs/00-status.md](../../stories/00-status.md)
**Scope:** the measurement backbone — a benchmark workload in `.wo`, a
campaign driver script, a tracked baseline contract, and the durability
proofs (restart persistence + crash battery), single- AND multi-shard.
**Relates to:** [story 22](../../stories/language-runtime-database/in-progress/22-durability-throughput-scale.md)
**Relates to:** [story 22](../../stories/language-runtime-database/done/22-durability-throughput-scale.md)
(the four forks settled below), the landed arc
([story 8's guarantee contract](../../stories/language-runtime-database/done/08-shard-actor-runtime.md)
— the stage-3 delta this iteration records), iteration 23 (the durable