databasev2 4 part A, task 6. Mostly documentation, plus one real fix the
full battery caught.
THE FIX. The drain held EVERY DB reply until the barrier — including
reads, which stage nothing and have no stake in durability. That parked
readers behind an fsync for no reason: durable.sN.mixread.p99 rose from
~1043us to 4057us. Only a statement that actually staged a record now has
its reply held. Caught by the gate, not by review.
THE TRADE, recorded rather than smoothed over. What remains is inherent: a
barrier blocks the owner shard LONGER (more records per fsync) though LESS
OFTEN, so anything queued behind one waits. Three full runs of the same
build gave durable.sN.mixread.p99 of 1043 / 2318 / 4147us and wmix.p99 of
8758 / 20000us — a 2-4x spread with the box near idle. So part A buys ~3x
write throughput at the cost of a longer, noisier tail on the owner shard,
and that is the strongest argument for part B (submit and keep serving).
- durable.sN.*.p99us tolerance widened to 100% WITH the reason in the
code: a 2-4x-variable tail gated at 50% gates the disk, not the engine.
The floor is the real guard and is not slack — mixread's (4172us) came
within 25us of tripping on the worst run. Baseline refreshed; a fresh
full run then passed 106 checks 0 failures
EXIT STATUS MOVED 3 -> 74 (sysexits EX_IOERR). 3 and 4 are already used by
SAMPLES for their own meanings — db-bench's own `verify` exits 3 on a
checksum mismatch, and it is the gate that exercises durability, so a
durability abort exiting 3 would have been indistinguishable from the
mismatch it should help diagnose. The low range belongs to programs.
Docs:
- story: progress, the payoff measured two ways, the cost side, criteria
split met/outstanding, and a "part B — its premise changed" section:
it was justified by "close the 66x gap", but that gap is two problems
and only the concurrent one was a batching problem
- board: standup entry in the six-question shape; both databasev2 4 rows
rewritten. They had said "close the 66x gap" — recorded as MIS-STATED
rather than quietly renumbered
- 00-wob-format.md and 04-db-binding.md: the normative failure contract
("a failed WAL commit traps WO_T_IO after un-applying the row") was
false; corrected, along with the tick-scoped group commit that never
happened
- database/src/CODE-LOGIC.md: where the barrier runs and why there, why
replies are held, why the inline path is asymmetric, the one failure
rule, and how to measure it
- db-bench README: the wmix mode, the env knobs, and the tmpfs warning
Battery: wovm-test 36 suites 0 fail, woc-test, oop-e2e 119/0,
db-bench 106/0, linkcheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
databasev2 4 part A, task 5.
Controlled before/after — same machine, same workload (wmix 4000 32),
same build except db.c and vm.c, two runs each interleaved:
- per-statement barrier: 2213 / 2177 ops/sec, p50 7183 / 7251us
- group commit: 6216 / 6525 ops/sec, p50 3458 / 3444us
- ~2.9x throughput, ~2.1x lower p50
The full campaign confirms it a second way: s1 takes the inline path and
commits per statement BY DESIGN, so within one build the shard configs are
batching-off vs batching-on — 1467 -> 5117 ops/sec, mean batch 1.0 -> 5.43,
peak 1 -> 57. 3.5x, agreeing with the 2.9x above.
Recorded honestly:
- the BEFORE p99 is at the histogram ceiling (hist_add clamps at 20000us
and both runs pinned there), so the true figure is >=20ms and unknown.
The improvement is AT LEAST 2.3x; the old p99 was off the instrument
- durable.sN.mixwrite went 480 -> 492 ops/sec, i.e. UNCHANGED. That was
the spec's original payoff metric and correcting it was part of the
brainstorm: mix performs 20 writes at C=4, mean batch 1.01. A workload
that never has two writes in flight cannot be helped by batching them
- seed is likewise unchanged: a serial writer has nothing to batch with
- so the payoff is real but CONDITIONAL — it appears where concurrent
durable writes fan into the owner shard, and nowhere else
Two traps recorded in perf-targets §6:
- do not benchmark durability on /tmp: it is tmpfs here, where fdatasync
is free. The same run reported 195000 ops/sec at p50 1us there against
2200 at p50 7200us on ext4 — no barrier to amortise, so the measurement
measures nothing. db-bench keeps its stores under bench/ for this reason
- the record count is not the update count: 7755 records for 4000 updates,
because hist_dump and the done-marker are themselves durable inserts
- FIXED a regression I introduced in T4: master's committed baseline is
FULL mode (N=20000, crash_reps=3, msg_n=200000) and I had overwritten it
with quick-mode values. Regenerated from a full campaign; the full run
now passes 106 checks 0 failures against it
- gate still bites: sN wmix ops_sec -70% -> FAIL on exactly that metric
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
databasev2 4 part A, task 4. Scope extended with developer approval: the
plan authorised touching the sample only for observability, but no
existing leg has enough concurrent durable writes to exercise group
commit at all, so the payoff was unevaluable either way.
The finding that forced it:
- `mix` writes on one op in ten with C=4 (all_mode calls mix_mode(n/10,
4); Mixer writes on i % 10 == 9), so the quick run performs 20 writes
total. Measured mean batch 1.01 over 3112 barriers, peak 3
- that is a property of the WORKLOAD, not the mechanism: peak 3 of a
possible 4 shows batches form whenever writes actually coincide
- `wmix N C` added: every op a durable write, C at once. Updates rather
than inserts, so it is comparable to mixwrite and the row count stays
flat. Histogram kind 2 — a replayed store still holds the seeding run's
kind-0/1 Hist rows and merging those would report someone else's
latencies
- WO_WAL_STATS=1 prints one line at exit: batches, records, peak_batch,
peak_staged. Opt-in, because it would otherwise pollute every durable
program's output. Counters live in wo_wal; no builtin, the numbers are
diagnostic and not part of the language
Measured, and it scales with concurrency exactly as designed:
- C = 4 / 16 / 64 -> mean batch 1.13 / 1.76 / 5.35, peak 3 / 10 / 39
- the gate's own legs: durable.s1 5412 records over 5412 barriers (mean
1.0, peak 1 — the inline path, one barrier per statement BY DESIGN),
durable.sN 7757 over 2296 (mean 3.38, peak 28) at 2x the throughput
- peak staged 1372 B settles the no-cap decision with a number: the batch
is tiny, so the upstream mailbox bound is sufficient
- mean_batch/peak_batch are higher-is-better (the default detector would
have called bigger batches worse)
- only the batch SHAPE metrics are waived to 100%; wmix throughput and
latency keep real tolerances (15% s1, 50% sN) — a blanket waiver would
have left the entire new leg ungated
- the live assertion `mean > 1.0` on the sN leg is what catches inertness
- gate bites: sN wmix ops_sec -60% -> FAIL on exactly that metric, 1 of 86
Verified: db-bench-quick 89 checks 0 failures; baseline refreshed (86
metrics).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- engine: wo_idx_probe answers single-column equality from the index
hash buckets (idx_hash_key1 reproduces idx_hash bit for bit; verify
compares exactly as the slab walk did, so results identical);
composite indexes keep the walk; both executors wired (local + DB
actor RPC)
- compiler: probe_key_of_where lowers "var.col == key" on an indexed
column to DB_PROBE; all where guards still run (guard stays the
final arbiter); keys = ident/int-literal only; Float/Bytes excluded
(engine raw-eq narrower than VM float-eq)
- measured: reads 1.3k -> 1.3M ops/s, p50 600us -> 1us (~x850);
query x830; mixread 1.3k -> 89k s1, 21 -> ~1.9k sN
- gate policy moved into the driver (tolerance_for: refresh-proof);
latency floors max(4x,100us); quick mode skips poll-bound mix
floors; both tolerance classes proven to bite
- proof: test_table wo_idx_probe suite (RED first), corpus
query-index-probe 105/0, full battery green, TSan clean, two
campaigns pass the refreshed baseline
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- bench/baseline.json: 74 metrics from the first full campaign;
tolerances tuned by a two-run repeatability check (mix* 50%,
read/query 35%, rest 15% — rationale in _config)
- gate bites: --check mode; doctored copy fails, both real runs 74/0
- headline: durable seed 4.5k/s vs ram 297k/s (23's case); reads
O(table) at ~1.5k/s; mixread 1280 vs 21 ops/s single-vs-multi
(the arc's price); msgrate 13.4M vs 2.45M (mutex-inbox number)
- arc delta recorded in story 8; findings in sample README
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
