- jarvis 1 (chat loop) brainstormed to ready with forks AUTO-APPROVED for
autonomous execution and flagged in `review_pending` frontmatter for the
developer's second review: Anthropic Messages API backend, env-var API key,
actor-per-conversation SSE relay, durable @table history, session-gated routes
- jarvis 2 (tool use) + 3 (retrieval/RAG) created at refine with forks named
- 00-story iterations table linked to the new files
- blocked until rv2 9 TLS reaches phase F; pure .wo on porch 2/3/6/7 + the seam
(cherry picked from commit 8e160c3fcf9c50a05af073c036c693b192c9e595)
- portable constant-time AES-GCM software path; AES-GCM now on any CPU
(hw-or-sw dispatch), NIST-KAT-gated both paths (48/0). Remaining D/E;
ARMv8 hw path deferred. Board synced
(cherry picked from commit 138de17988e6bd274abe5e1e2d444ff2689747cd)
- phases A + B done; B = AES-128/256-GCM via AES-NI/PCLMULQDQ, NIST-KAT-gated,
ASan clean, portable binary (CPUID-gated). Phase C now owns the software
fallback AND the ARMv8 hardware path (deferred, untestable on x86-64 host)
(cherry picked from commit 249b1dbd72122ed6c05f4f0aadb7e1a5e87f844c)
- a second consumer (rv2 9 TLS phase A) reshaped the forks since the draft
- locked: BOTH AES-GCM (128/256, TLS-mandatory per RFC 8446) AND
ChaCha20-Poly1305 (RFC 8439, easy constant-time, cookie default)
- AES constant-time via AES-NI/ARMv8 hardware + bitsliced software fallback
(compiler intrinsics, zero external dep)
- caller-supplied nonce (TLS builds its own per-record nonce); random-nonce
is a cookie WRAPPER (phase D) not the primitive. shape:
seal(key,nonce,aad,pt)->Bytes / open->?Bytes; AES variant by key length
- raw key + length check; hand-rolled (matches rv2 9); ids from 111
- phases A ChaCha -> B hardware AES-GCM -> C software AES -> D cookie wrapper
-> E gate (RFC 8439 + NIST GCM vectors, ASan, reference cross-check)
- risk/test: constant-time mandatory, KAT-gated, reused-nonce documented
- retires the stale "TLS proxy-terminated" OOS line (rv2 9 overturned it)
(cherry picked from commit c8a5a31c39a5d14952056cd0af1b8c1e42d4ed2d)
- decision: HAND-ROLL TLS 1.3 (no vendored lib) per developer call; keeps the
zero-external-dep single binary, and raises risk rather than lowering it —
recorded, owned, with mandatory mitigations
- 1.3-only; RSA-PSS/PKCS1 + ECDSA-P256 + full ASN.1/X.509 chain validation +
trust store + hostname (the scope needed to reach real LLM APIs)
- decomposed into a bottom-up phase ladder: A AEAD (=rv2 8, forces AES-GCM
there) -> B HKDF -> C X25519 -> D signatures/RSA -> E X.509 -> F record+FSM
client -> G inbound server; C/D/E may each split into own iterations
- risk + test strategy section: constant-time, reference-tested (openssl +
RFC 8448 vectors), negative tests first-class, no partial-trust states
- deps: rv2 8 (AEAD), lang 34 (SHA/HMAC), net.connect (110, landed). Board synced
(cherry picked from commit f1881cca8cd0cdd58b1ac844e3e3ea9c234bb99c)
- jarvis (00-story): 6th track, 2nd software built with writeonce — an AI
assistant; direct-HTTPS design; blockers named (net.connect + TLS)
- runtime-v2 7 observability + 8 symmetric cipher: moved from the language
track (were 30/43); 9 in-process TLS: created from the gap jarvis surfaces,
RETIRES the "TLS is the proxy's job" doctrine (both directions)
- language 41 (arena hang): fix design to ready — marshal cross-shard
messages (root), align the shard_id % nshards route/compare + assert bound;
poison-on-free + minimal fixture as follow-ups
- fiber scope-gap analysis (plan/exploration/fiber/01): porch vs fiber, what
porch lacks, would developers prefer porch
- board + dependency-graph synced (porch 2-8 ready; rv2 table; §5/§5a graphs)
(cherry picked from commit 203470ceb2a151fe3584931cd4237af3f96a9f29)
- whole porch track (2-8) now brainstormed and locked (all ready)
- four decisions: three hooks (on-listen/on-shutdown/on-route-registered);
healthcheck ships BOTH /livez + /readyz; directory listing off-by-default,
documented; Last-Modified via a new small time.utc(ms)->TimeParts builtin
- language enhancement: YES, one small builtin -- time.utc, a gmtime sibling
of time.local (time.local is local-tz, time.iso is UTC-but-ISO); IMS by
string-equality, no date parser. The track's third + smallest language touch
- byte ranges/large files via fs.read_at + iteration 6 writer; not lang-41-exposed
- track language bill now explicit: random_bytes (2), deflate+crc32 (7),
time.utc (8) -- each a builtin with a named consumer, none decoration
- validated against .dev/reference/fiber. Board: whole track marked ready
(cherry picked from commit 9801fceade799e25718606177f09e4306a579e98)
- five decisions: refuse incoherent heartbeat/idle_ms pair at construction;
codec = two C builtins deflate+crc32 (perf over pure-.wo; hand-rolled, no
zlib dep; gzip framing in .wo); ETag over uncompressed bytes + Vary;
Last-Event-ID explicitly unsupported (not silently ignored); Vary via
comma-join
- language enhancement: YES, two builtins -- the track's SECOND language
dependency after iteration 2's random_bytes. CRC32 finally gets its
consumer; inflate deliberately not built (request-body decompression OOS)
- corrected stale dependency: Vary uses iteration 5's comma-join, so story 7
depends on 6 + 5, NOT 2; codec is pure compute, not lang-41-exposed
- confirmed CRC32 absent + iteration 36 bit operators landed (pure-.wo was
viable, traded for hot-path speed)
- validated against .dev/reference/fiber. Board synced
(cherry picked from commit 07f53574dd90f502235b79d4920eab3d684c8b77)
- re-scoped to OUTBOUND streaming only
- three decisions: separate StreamHandler/BodyProducer parallel path (Resp
path untouched -> existing responses byte-identical); streaming routes opt
out of the after-chain, framework refuses at registration to combine with
header-mutating middleware (loud, never silent), security_headers() helper
lets handlers stamp them; chunked REQUEST bodies split into their own future
iteration (parse.wo refusal stays, smuggling cases enumerated for later)
- no language enhancement (net.write framing, fs.read_at/actor source,
interfaces for producer); rides the fiber loop not the actor pool, so not
lang-41-exposed
- fixed title inconsistency: "three iterations wait on" -> "two" (7 and 8)
- validated against .dev/reference/fiber + the app.wo/serve.wo pipeline. Board synced
(cherry picked from commit 15205408e03c3c02a38e00e5d2017a8a11f62f28)
- five decisions: head auto-registers with opt-out (+ patch/options/all);
request ids mirror limiter trust model with a NON-crypto source; per-route
body_limit is a SECOND check after routing (global BODY_MAX stays the
pre-routing ceiling, over-limit = 413); Route fields are corpus-free;
Vary accumulates by comma-join
- key finding: story 5 has NO upstream dependency, not even iteration 2 --
request ids are not secrets, so a non-crypto source (time.ticks+counter)
keeps it startable today; the one porch slice buildable right now
- three story assumptions corrected: per-route limit cannot replace the
global (body read before routing); the container-owned-move corpus fixture
has its OWN Route (adding fields is free); Vary needs no iteration 2
- validated against .dev/reference/fiber; zero language enhancement. Board synced
(cherry picked from commit 0589a13db1f3b7c220d9d9fdc76142af6a63c390)
sessions (3):
- six decisions: pure-auth-primitive row (no payload bag); wall-clock
time.now not monotonic time.ticks (restart durability); login always
mints a fresh id (fixation, no anon-session model); throttled last_seen
touch at idle/20 (not a WAL write per request); Session writes
req.principal; config refuses absolute < idle
- finding: no per-key actor pool, so NOT blocked on lang-41 (plain @table
CRUD, same path storefront uses); the no-bag rule closes the one place
fiber's Set(key,any)+msgp+RegisterType would have hit principle 13
csrf (4):
- five decisions: fiber's hybrid transport (session-stored CsrfToken
@table + double-submit cookie, both must pass; no CSRF for sessionless
apps); opt-in single-use (checkout example); double-click -> distinct
SPENT refusal, NOT coupled to lang-41-blocked idempotency; trusted
origin/referer/Sec-Fetch-Site second layer; refusal classes distinct in
logs, opaque in body
- no actor pool, not blocked on lang-41
both validated against .dev/reference/fiber (v3, 3ca9a9d); exactly ZERO
language enhancement needed beyond iteration 2's random_bytes. Board synced.
(cherry picked from commit 3a4fb4215b23d2516362e2dd0acc5bec6c9aebc0)
- five forks locked: cookies: multi SetCookie beside unchanged headers
map; bare-name random_bytes(n)->Bytes; structural-400 in parse_request
+ on-demand cookie() helper; base64(value).base64(mac) signing;
app-supplied key, no middleware (that is iteration 3)
- validated against .dev/reference/fiber (v3, 3ca9a9d): exactly ONE
language enhancement needed (the CSPRNG); repeated Set-Cookie, cookie
attributes, parsing and signing all map to existing primitives
- corrects phase A registry: random_bytes joins the crypto-family
bare-name table (emit.ml b_* + types.ml), NOT wob.h's module enum;
next free id 84/90, not 110
- board: story 2 marked ready, porch-2 row rewritten off the stale
wob.h/110 claim
(cherry picked from commit 4d31d5359436496aed40cb25611abc7ccd4d7875)
- five stories status: done; 00-story records the one-run landing
- spec History: three implementation amendments (Signal record not
scalar, caller-owned stdio fds, handler-latch instead of signalfd)
- board NEXT PLAN entry with measured findings (zero transport code
added; the tty-across-the-socket handover proven; the double-raw
refusal restoring the terminal — the "bug" that was the design
working); section rows flipped; graph nodes green
- CODE-LOGIC.md: the runtime-v2 section
- full belt quoted on the board: suites 0 fail both flavors (test_proc
193/0, test_term 60/0), woc 557/0, subprocess 12/0, site 23/0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit bc1b4f070693eb755ad6a9fd0c853fb3e2bda347)
- spec 2026-09-01-runtime-v2-design.md: the one principle (PULL — a
child is fds, the net verbs drive them; runtime-v2 adds acquisition
verbs, never transport), the full surface (ids 97+: spawn/spawn_pty/
wait_dl/signal/resize, signal.on delivering the sig number, term.raw/
restore with runtime-guaranteed restore, send_fd/recv_fd/connect_unix),
actor-owned lifecycle, mechanics notes, refusals by name
- push transport rejected with reasons recorded (mailbox-cap collision,
new delivery machinery); death-notice verb refused (a two-line fiber
composes wait_dl)
- five stories flip readiness: ready; fork sections rewritten as settled
- graph section 6 remapped: pull broke the 1->2->3 chain — only 1->2
remains; 3, 4, 5 and the VTE grid startable alone today
- board section + registry follow; linkcheck 0 broken
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit d313cdeebbe53c83b83f31f4480631568d9d0743)
- root cause: a worker's runtime is initialised lazily on first fiber
adoption, and rt.shard_id is stamped only there — but INBOX_READY[i]
is set at thread creation. A shard that never adopts is still settled
at shutdown, carrying rt.shard_id 0 from the memset
- it then impersonated shard 0: wo_drop_obj saw 0 == 0 for anything the
primary allocated, took the "we are home" branch instead of routing,
and called class_free against rt->classes, which lazy init never
filled. &rt->classes[class_id] off a NULL base is the faulting read
- fix: stamp the runtime's real identity at thread creation. An
uninitialised shard owns nothing, so its true id makes every payload
correctly foreign and routes it to an owner that can free it
- ASan could not name this: the arena is one hand-managed malloc block,
so intra-arena reuse is invisible and it surfaces as a bare SEGV
- pinned by tests/regress/lang-41, driven from db-actor-accept. Needs
multiple shards (the corpus runner pins WO_SHARDS=1) and the ASan
build. SEGVs twice per run unfixed, clean fixed
- the HANG is a separate defect and is NOT fixed: with this in place the
harness stops losing whole sections, but idempotent-stop-2 still
fires ~1 run in 6. The story records where to look
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 9dca0b4b4727b976d326b29cb4c6522b62d48a73)
- Add saturation leg (scripts/web-app-accept.sh): one-actor pool,
WO_MAILBOX=2, 15 concurrent requests, exactly 3 served + 12 answer
503; execution count matches the 200 count, retry-after + real
cause verified on the 503s
- Guard make_pool(n<1) by clamping in make_pool itself, not
pool_select's division -- that trap runs inside the middleware's
own try/catch and would be swallowed as ordinary saturation forever
- README: rate limiting + idempotency ledger rows moved to done,
scoped to what the gate proves; documented Handler-decorator
shape, Pool aliasing (WO-E222), call's scalar-only reply (WO-E226),
pool size as a capacity decision
- Story: Progress table filled with real hashes, 7/9 acceptance
criteria marked verified with citations, 2 marked verified by
construction (never gated even in the original plan), status: done
- Status board: standup entry, porch 1 pending row updated
- Recorded a pre-existing runtime hang (main() returns cleanly, OS
process sometimes hangs under concurrent call()-parked callers)
that also reaches the new leg's teardown; contained with kill -9
rather than asserted, so it can't flake the leg's actual subject
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 21934b18910070a3f24b8bd4367fcb9d397dc1fb)
- resolved 9g's forks EMPIRICALLY against the running compiler:
- count(<query>) and len(<query>) already work (fork 2 collapses to
zero code)
- skillhost's correlated NOT EXISTS is a backlink emptiness in
writeonce (`where len(x.children) == 0`), using only 9b machinery
(fork 1) — verified on a self-referential ?ref/backlink table
=> corpus #1 forces NO new grammar; per the method ("add only what a
corpus uses"), exists/not-exists was NOT built
- docs/examples/skill-catalog: mirrors skillhost's `skills` table
(name @unique, description/location/root, parent ?ref Skill, children
backlink) and translates all five of its SQL statements 1:1
(insert+dup-trap, get-by-name, list, roots via backlink-emptiness,
count); scripts/skill-catalog-accept.sh 7/0, WAL-durable, dup trap
persists across restart
- fixture run/db-query-corpus (count(query) + backlink NOT EXISTS);
just skill-catalog module; target/ gitignored
- general exists/not-exists left unbuilt and recorded as "enters when a
corpus forces a non-relation correlation"
- gates: oop-e2e 80/0, woc-test 566/0, skill-catalog 7/0; story + board
record the finding
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 4c82461634d45f11eca1a252702031c03d999c7f)
- story frontmatter status: done, Progress section records what landed
vs the spec (everything, same day as the brainstorm)
- board: NEXT PLAN entry with the six standup answers (deadlock proven
real: 5 s hang, 8192-byte truncation; 15 ms after; ping 2 ms during a
parked child; 1000 spawns fd-flat; SIGTERM leaves no child); pending
row flipped to DONE
- graph: node 42 class done, same change as the board row
- runtime/src/CODE-LOGIC.md: the bounded-subprocess section (bundle
park, slot registry, ownership sweeps, raw pidfd syscalls)
- full belt at close: 19 runtime suites 0 fail (test_proc 128/0),
woc-test 557/0, subprocess-accept 12/0, site-accept 23/0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- crash-before-rename test: a COMPLETE valid migrated temp beside the
untouched original is discarded and the boot re-migrates — the
sharpest point on the crash timeline, deterministic, no fault
injection needed
- story: all six tasks done with commit hashes, all eight criteria met
with the test that proves each, plus the three deviations from the
plan and why (transcode over replay, lazy head, poison forces
transcode)
- CODE-LOGIC: migration section; also corrected limitation 3, which
still claimed unbounded hot-row chains — iteration 11 closed that
- status board row 12; deploy guide's rollback section gets its real
answer (rolling back across a migration is a migration backwards:
expect the refusal, restore the .bak)
- test_wal 5966 pass, 0 fail
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 4bb6ece2531e2123eb958c91d9bef4a6528eab3b)
- brainstorm settled: declarative and automatic at boot; v1 verbs are
add and delete only; data/seed migrations deferred to v2
- added fields zero-fill by kind: the grammar has no field-default
syntax and v1 refuses to grow compiler surface for it
- same-kind delete+add refuses as a disguised rename; retype and
vanished classes refuse by name
- schema lives in the log itself: WO_WAL_SCHEMA head record, written by
fresh-log open and compaction; name-keyed diff also closes the
silent cid-renumbering hole
- story is iteration 12, board row added, db2-migrate prefix claimed
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 072e007b144ff6689b6ff920ea10665900c1a2ef)
- status: done in the story frontmatter (was the non-conventional
"complete"; the board's axis uses done/in-progress/pending/hold)
- board row 11: what landed, the WO_WAL_UPDATE correction, the ceiling
removed as unreachable, and the one criterion still weaker than
written (expected value, not a resident: all oracle)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit de39a88e81e97bf6f8b75e9f15331aae20f7ab29)
- flattened row image is WO_WAL_UPDATE, not WO_WAL_INSERT: the row's
original INSERT is already in a live log, so a second one for the same
id is a duplicate replay refuses as corruption. INSERT is right only
for compaction, which builds a fresh log
- remove WO_CKPT_MAX_GARBAGE: with the absolute term at 64 MiB, garbage
large enough to reach a 256 MiB ceiling has already tripped it, so the
branch was unreachable. Postgres needs both constants because it
thresholds on tuples with its pair at opposite ends; this thresholds
on bytes, where one constant does both jobs
- test_delta_chain_flattens_at_k: chain depth stays <= WO_DELTA_MAX_HOPS
across 2K+2 updates, and a reset is observed
- test_delta_chain_flatten_replays: a flattened chain replays correctly
- test_keys_resident_indexed_across_flatten: a delta on an indexed
column composes with flattening, checked at every step across the
bound and after restart. Found no product defect
- test_should_compact_absolute_and_ceiling: pins the absolute term, the
boundary just under it, and the small-log case the ratio still governs
- test_wal 5700 pass / 0 fail; wovm-test and woc-test green
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f93b5d9db753305c297e868d977670e6d703684c)
TESTS DELIBERATELY HELD at the developer's instruction — logic only.
The existing suite passes (36 suites, 0 fail) but exercises NEITHER new
behaviour: nothing builds a 16-deep chain, and no checkpoint test uses a
log near 64 MiB. Green here means "did not break what existed".
- tier 1: wo_wal_fold_row_at gains hops_out. The walk already visits
every hop, so the depth is free — this is the design's pd_prune_xid,
a cheap "is work worth doing" hint taken from work already happening
- the update path branches on it: past WO_DELTA_MAX_HOPS (16) it writes
a full-row image instead of a delta, terminating the chain. `r`
already holds the complete post-update row because index maintenance
required folding it, so flattening costs bytes, not an extra read
- wo_wal_append_row_image encodes from a caller-held row, as
WO_WAL_INSERT: a chain's base must replay into a database where
nothing precedes it, so replay/compaction/fold need no change
- tier 2: should_compact gains a TRIGGERING absolute term and a ceiling.
Our `floor` SUPPRESSES on a small log — the opposite of postgres's
vac_base_thresh, which triggers on a small absolute problem the
proportion hides. We had the proportion and the suppressor and
neither real guard
- verified by construction, not test: both update entry points converge
on row_apply_field_keys; db.c captures next_offset BEFORE calling in,
so the re-point is transparent to which record type was written
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 1b808abd5942de81c3a6416714d1302384103040)
- new `residency` leg in db-bench.py driving docs/examples/residency-bench:
two tables identical except the annotation, control cap + binding cap
- ITS OWN PROGRAM, not a db-bench mode: declaring a resident: keys table
is a WHOLE-PROGRAM constraint, so the no-WO_DATA refusal fires for
every mode in the module. Putting those classes in db-bench's shared
types made growth/ceiling/randread — which run without WO_DATA —
refuse to start. Caught by running the leg, not by reading it
- gates the RATIOS, waives the absolutes: ops/sec under a cap is swap
and disk I/O and belongs to the box. Same split randread makes
- rss_ratio 2.55 floor 2.0 tol 10% (structural, like bytes_per_row);
overcap_vs_swap_x 1.53 floor 1.0; in_ram_cost_x 4.23 ceiling 8.0;
all_collapse_x 105.4 floor 2.0
- all_collapse_x exists because the leg's FIRST run silently measured
nothing: at QUICK's 40k rows a 48 MiB cap binds neither mode, so the
"over-cap" half was not over cap. The cap now scales with N and the
leg asserts it binds
- verified the gate bites: rss_ratio 1.4, overcap_vs_swap_x 0.6 and
in_ram_cost_x 12.0 are all rejected
- task 7 closed: both criteria moved to Met with how each was verified
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a310496664982c372f51b43113465eb8ad9e9fb5)
- two tables identical except the annotation, 200k rows, 40k reads in
one key order, WAL on ext4 (not /tmp, which is tmpfs here and would
have put the log in RAM), rootless cgroup v2 cap
- WIDE shape, 2.55x smaller resident set: 34.4 MB vs 87.5 MB. That is
the real win and the thing the mode was built for
- under a 48 MB cap (between the two resident sets): keys 19635 ops/s
vs all 12854 — only 1.53x faster than letting the kernel swap
- degradation is far gentler though: all collapses 105x from its own
uncapped throughput, keys 16x
- costs 4.2x read throughput when memory is not tight, and writes are
markedly slower — the keys fill did not finish in 2 min where the
resident fill plus 40k reads did. No design doc had costed writes
- THE UNANTICIPATED FINDING: cgroup limits charge the PAGE CACHE, so
moving rows to a file does not escape a container memory limit. WAL
37 MB + RSS 34 MB cannot both live under a 48 MB cap, so every pread
reaches disk. The premise "the page cache will hold the hot rows"
fails in exactly the deployment this targets
- first attempt used Int-only rows and showed parity; recorded, because
drop_payload frees a field's VALUE and an Int's value is its inline
slot word, so that shape cannot benefit and would have condemned the
feature for the wrong reason
- verdict: keep it, to fit ~2.5x more data in given RAM — not to make
an over-capacity table fast
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 7cba9b1174b0bf581314b3147e25cc49e6f49464)
- fixes a limitation iteration 2 shipped: compaction was supposed to
bound chain length, but wo_wal_should_compact triggers on a whole-log
byte ratio and cannot see one hot row's chain
- tier 1, flatten on update: the update path ALREADY folds the row for
index maintenance and the fold already walks hop by hop, so it reports
depth for free. Past a fixed K it writes a full row instead of a
delta. Read <= K+1 reads, replay O(K^2) per row. No format change, no
per-row RAM, no new trigger
- tier 2: our compaction policy has a proportional term and a
SUPPRESSOR misleadingly called a floor; postgres's floor TRIGGERS on
small absolute garbage. Add that term and a ceiling
- design read from .dev/reference/postgresql, not recalled:
heap_page_prune_opt gates on an O(1) on-page hint then page fullness
against Max(fillfactor, BLCKSZ/10); autovacuum uses base + scale *
reltuples clamped by a max (50, 0.2, 1e8). Neither thresholds on
new-bytes-versus-old-bytes
- K deliberately does NOT scale with table size: postgres scales a
table-level aggregate with proportional harm, ours is per-row with
additive cost, so scaling up would make big databases boot worst
- the story says plainly it should NOT be next: task 7 has still never
measured whether resident: keys beats the kernel's own paging
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f667cad2cfbe187b5973440ab1af015b2df288f8)
- "no storage behind it" / "nothing yet stores a table that way" was
false — CRUD, checkpoint survival and updates all landed; replaced
with an accurate summary naming task 6/7 as what remains
- the three checked delete/delete-replay/update criteria sat in
Outstanding despite being done; moved to Met, leaving Outstanding
holding only genuine task 6/7 work
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit b575678ee95fa3125fa4e8145320a9cdca10ba38)
- loader.c: delete the INCOMPLETE-update BAIL; durable:false +
resident:keys stays refused (nowhere to read from)
- table.c: root-cause fix for the Text-index gap — a keys-resident
borrow now holds ENGINE values, matching wo_row_ptr's contract
(table.h's "no VM pointer" doctrine), not a VM-decoded row. Fixes
idx_hash/idx_cols_equal/wo_idx_probe AND db.c's GET_FIELD/PROBE
arms with one change; reproduced pre-fix as an ASan
heap-buffer-overflow
- docs/examples/residency: Product is genuinely resident:keys;
residency-accept.sh's refusal leg replaced by proving the program
runs and stock survives a restart (11/0)
- test_wal.c: oracle test drives resident:all and resident:keys
through the same update sequence and asserts identical rows;
Text-indexed-update test catches the representation bug; five
pre-existing tests corrected to the fixed contract (4746/0)
- story, README, status board, CODE-LOGIC.md updated; three known
limitations documented: mid-drain stale reads, O(N^2) replay in
chain length, compaction blind to per-row chain length
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit b87c68f950f01aa5e572fbb86a0f374adc83d813)
- wo_row_remove's keys arm borrows the row from the log to find its
index entries, and a borrow reads through db->rt->wal. At boot that
pointer is not wired yet: main.c replays first (main.c:226) and
assigns rt.wal afterwards (main.c:268)
- so the borrow found no log, the remove failed, and replay reported a
valid tombstone as CORRUPTION. An UPDATE record would have failed the
same way, since replay applies it as remove-then-recreate
- replay now lends the runtime a read-only view over the fd it already
has open, for the replay's duration only, and restores what was there
- broken by the delete fix in 76b8fd9 — deletes worked in-process but
their tombstones broke the next boot. Unreachable in production only
because the loader still refuses the annotation
- pinned by test_keys_resident_delete_then_replay, verified failing
against the unfixed code (2 failures) and clean with it
- found by asking whether the read-modify-append plan was ready, not by
a gate
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit dc25462461b9f79d70c803f7174adc90fa16c90e)
- docs/examples/residency: one program, two tables filled by the same
loop, differing only in the annotation. Run twice against one WO_DATA
and orders replay while sessions do not
- the example checks its own claim (exits 1 if a durable:false table
survives, or a durable:true one fails to replay) rather than narrating
it in a print
- resident: keys is written out as a commented block with the loader's
exact refusal, so the frontier is visible in the example rather than
only in a story. It documents WHERE the refusal happens: woc compiles
it and emits a .wob; wovm exits 2, because the annotation is a
load-time property
- residency-accept gains two legs: the example runs and its restart
claim holds, and the refusal message the README quotes is checked so
doc and code cannot drift apart
- the gate writes the example's output to /tmp/residency.log,
banner-separated, for tail -F
- README commands verified verbatim; they needed mkdir -p because wovm
will not create WO_DATA
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c9c7e03e62c3918cb65ef7994d1e33a0c5337b71)
- wo_row_remove read the id map's value as a slot, but on a keys table
that value is a LOG OFFSET (hput(t, id, wal_off + 1)). slot_row does
no bounds check, so a delete indexed t->slabs[] with a byte offset and
then called db_val_free on whatever it landed on — arbitrary frees,
not a wrong answer
- keys tables now take their own arm: no slab slot, no bitmap bit, no
free-list entry to return. The index hook needs the row's values, so
the row is borrowed from the log for exactly that long
- wo_row_ptr carried the same trap and is public. It cannot refuse keys
tables outright (insert legitimately calls it while the map still
holds a slot), so it now detects the offset case — index past the
slabs, or bitmap bit clear — and returns NULL. Callers all handle NULL
- test_keys_resident_delete pins it; it SEGVs against the old code,
verified by reverting the fix rather than assumed
- found while auditing every hget() reader before narrowing the loader
refusal to allow benchmarking. The refusal was justified in the docs
by "updates are unimplemented" while actually standing in front of
this too: a guard whose stated reason is narrower than its real one
gets removed by someone who believes the stated reason
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 76b8fd944af9ed062467bdf9ab93c2e96dd198cf)
- loader's resident:keys refusal said "rows are still fully resident"
and "until tasks 5c/5d land". Both false since f606fc9. Corrected to
name the real blocker: UPDATE needs read-modify-append
- databasev2 00-story: the sequence graph drew 2->3->4, which reads as
3 needing 2 and 4 needing 3. Both backwards, and it still drew the
2->5->6 path the 2026-08-27 amendment retired. Redrawn stating only
real dependencies, with 4 and 3 shown as composing rather than
ordered, and the execution order that actually happened
- databasev2 03: the hazard and its Outstanding entry both claimed
nothing fails "because iteration 2's storage half is unimplemented".
Marked discharged, and recorded that the hazard named only half the
danger — the bitmap walk would have dropped keys rows outright
- databasev2 06: pending -> hold (largely superseded, revisit only on
a measurement); dated its 5c/5d references
- porch 01: rewritten to the settled shape. readiness ready, status
in-progress, phases B and C marked superseded with why
- porch 01 claimed time.after "is still a reserved builtin id". False —
builtin 90, implemented. That claim is what made the iteration look
cheaper than it is
- porch README gains honest ledger rows for both features (partial,
being rebuilt), not shipped
- skill-catalog README pointed at a story path that moved tracks;
linkcheck now 0 broken
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit b3d8c403e1d19ac27ec966de85cb293e0765795c)
- wo_row_read and the @unique shadow probe go through borrow/release;
release runs before every exit, including wo_row_read's early return
- updates on a keys table refused explicitly in wo_row_update_field and
the slot variant: no slab slot to mutate, and writing the borrow's
scratch would discard the write silently. Needs read-modify-append
- compaction walked the bitmap, which a keys row has no bit in — every
such row would have been dropped from the new log. Now walks
wo_row_next_id and re-points each row to where it lands
- moves records byte-for-byte (copy_record) rather than decoding: a
borrowed row holds VM values, enc_val expects engine values, and ASan
caught that mismatch as a 4294967292-byte memcpy
- wo_row_set_offset updates a value in place and never rehashes, so a
wo_row_next_id cursor stays valid while compaction re-points
- a compaction that fails after moving rows is fatal: the map would name
an unlinked temp file, and the intact log replays correctly
- test_keys_resident_survives_compaction pins both failure modes; rows
rewrite in hash order so offsets really move
- loader still refuses resident: keys — updates are not implemented
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f606fc9b76d983cac2b348f03f7d4f01433cd905)
The branch was 17 ahead / 25 behind with 11 conflicting files, and drifting
further: db.c had been rewritten twice on master since (group commit, then
compaction). Resolved rather than rebased so both histories stay legible.
Conflicts, and how each was settled:
- db.c: BOTH semantics kept. Master's fatal path and compaction check now sit
behind the branch's `table_is_durable` predicate, in all three inline arms —
a volatile table reaches neither the barrier nor the compaction check
- db-bench sample: every mode from both sides (growth, growth-verify, randread,
replayseed, wmix) and ONE `boot` mode, which both sides had added
independently
- db-bench.py: all six legs kept. Both sides had also grown the same
WAL-size helper under different names; collapsed into one
- perf-targets: the branch's §5 (RAM ceiling) then master's §6/§7 — master's
numbering had already assumed a §5 it did not have
- story frontmatter: master's `status` (the landing truth) plus the branch's
`readiness` axis. 03 would have read `done` + `refine`, which is a
contradiction — it was brainstormed and landed on master, so `ready`
- board: both standup blocks newest-first; master's chain rows (a superset);
the branch's databasev2 1-2 rows with master's 3-4. Fixed a stray `|` in
master's row 3
- baseline: master's, then REGENERATED from a full campaign — 143 metrics,
132 checks, 0 failures with both sides' legs present
TWO HALF-EXPOSED FEATURES FIXED, because the merge rule is that master gets
no feature that is honoured in name only:
- `resident: keys` PARSED, set a .wob flag, and did nothing: rows stayed fully
resident. A developer could declare a 120 GB table keys-resident, watch it
compile, and be OOM-killed. The loader now REFUSES it with a message naming
what to write instead, until tasks 5c/5d land. The compiler still parses it
and its AST golden still passes, so the grammar work stays tested
- `durable: false` was honoured ONLY on the inline path. wo_db_exec_req had no
guard at all, so a volatile table written from an actor on a worker shard
would still be logged — precisely porch's session-table case, and precisely
what iteration 2 exists to provide. All three request-path arms now carry the
same predicate. Found by reading the merged code, not by a test: the obvious
probe runs main() on the primary and therefore only exercises the inline path
Verified on the merged tree: wovm-test 0, woc-test 0, oop-e2e 122/0,
residency-accept 8/0, db-bench 132/0, linkcheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
databasev2 3, task 6. Documentation, plus three gate-tolerance
corrections that are justified rather than silent.
- 04-db-binding.md: the NORMATIVE rule — compaction may run only where
nothing is staged (a correctness requirement, not scheduling), recovery
is unchanged, and a failed compaction is a missed optimisation rather
than a durability event
- database/src/CODE-LOGIC.md: why one file and not snapshot-plus-tail
(Postgres CANNOT compact — page deltas; ours are full row images, so a
compacted log IS a store), why rename is the whole crash-safety story,
why the dump flushes but does NOT fsync when it does, why the
replacement is preallocated, and where the trigger is checked
- README: the checkpoint knobs, the extended walstats line, the boot mode
- story -> status: done, with criteria split met/outstanding
- board: standup entry in the six-question shape, both rows rewritten
THE OBLIGATION IS AT THE COMPACTOR, not only in a spec: compaction moves
every record, so it invalidates every WAL offset iteration 2's
`resident: keys` stores, and the loop that knows each record's new
position must rebuild that map. Nothing fails today because that storage
half is unimplemented — it would fail later, looking like corruption.
Board claim corrected before it shipped: I wrote that the concurrency
chain is "complete". It is not — chain 5 stays in-progress because
databasev2 4's part B was never done and its premise was invalidated by
part A. Every link has landed its PLANNED work; that is a different
statement.
Gate tolerances, each with the measurement that justifies it:
- ckpt.pause_us_max is no longer gated relatively. The raw pause scales
with the live set and this workload's live set is not fixed (wmix's
hist_dump inserts a row per latency bucket), so gating it gates the
box. Added ckpt.pause_us_per_mb — the engine's own rate, gated for
real, and the metric that would have caught the 8x dump regression —
with the absolute 50ms budget still guarding the raw pause
- ram.*.msgrate 15% -> 70%. PRE-EXISTING, and measured: 10.7M-17.9M
msgs/sec across ten full runs, several predating this work — a 1.67x
spread against a 15% gate
- durable.sN.*.p99us 100% -> 300%, with more evidence than the first
widening: mixread 1043/2318/4147us, mixwrite 1623/4446us on the same
build. Floors stay the real guard and are not slack
Battery: wovm-test 36 suites 0 fail, woc-test, oop-e2e 119/0,
db-bench 117 checks 0 failures, linkcheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan for the approved spec. Code-free per the repo convention
(docs/plan/discarded.md:54); the executor writes the code.
- T1 wo_wal_compact: walk live rows via the bitmap, append one INSERT
each through the EXISTING append path, fsync, rename over the live log,
fsync the parent dir, reopen the descriptor. Test asserts BOTH that the
log shrank AND that a replay reproduces the same rows/ids/values —
shorter alone is worthless, a truncating bug also passes that
- T2 a stale temp file is removed at open and never read. The test uses
PLAUSIBLE records, not garbage: garbage would be rejected anyway and
would prove nothing
- T3 the trigger as a PURE decision (used bytes, last compaction's
measured output, floor) so it is unit-testable without a store; env
knobs for floor and ratio, which is what makes the policy testable at
all. No timer, with the reason. The check is called only where nothing
is staged, asserted by a test that stages and expects deferral
- T4 kill -9 DURING compaction, extending the existing fork-based crash
battery. Asserts the PROPERTY — the store equals the pre- or the
post-compaction content, never a mixture, and every acked id survives.
Run repeatedly and state the count: it is a race, one green run proves
little
- T5 measure space reclaimed, boot before/after, and the stop-the-world
PAUSE against a stated budget. If the pause exceeds it, stop and report
— the alternatives are bought against that number, not before it
- T6 closeout, including the normative ordering rule in 04-db-binding.md
Constraints carried from the spec into every task:
- recovery must NOT change; a task editing the replay path should stop
- the dump must FLUSH PERIODICALLY. stage() grows the staging buffer by
doubling, so dumping a whole store through one buffer would hold the
entire store in RAM — the unbounded growth databasev2 1 identified as
how this engine dies
- a FAILED compaction is a missed optimisation, not a durability event,
so it must not take databasev2 4's fatal path
- gate tolerances must not be waived wholesale (part A's T4 made that
mistake), and the baseline is full-mode — writing a quick-mode baseline
over it is a regression part A also made
Deliberately NOT a task: rebuilding the `resident: keys` offset map. It
cannot be implemented against a feature that does not exist yet, so T6
records it as an obligation at the compactor and in the story instead of
a stub nobody can test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
databasev2 3, chain 6. Brainstormed 2026-08-28 after databasev2 4 part A
landed.
Design: compact the log by rewriting it as one record per live row into a
temp file, fsync, rename over the live WAL, fsync the parent dir, reopen.
Recovery is COMPLETELY UNCHANGED — boot still opens one file and replays
it — and the crash criterion ("the same store as if the checkpoint had
never started") is satisfied by rename, not by code we must get right.
Read .dev/reference/postgresql for this. The finding is that PG's design
is UNAVAILABLE to us, which is what makes the simpler option legitimate:
- PG never compacts its WAL; segments before the redo point are recycled
by rename or unlinked. Its records are page deltas, so a compacted redo
log is not a store — hence heap files, a control file, a redo pointer,
a second recovery source and a separate process
- ours are FULL ROW IMAGES (apply_record implements UPDATE as
remove-then-recreate), so a compacted log IS a complete store. That one
difference deletes all of the above from the design
- what IS worth porting is the ordering discipline: publish the new
"recovery starts here" atomically and LAST, so a crash falls back. PG
needs a start-of-checkpoint redo pointer plus an end-of-checkpoint
control file update; we get the same property from one rename, because
we can swap the whole data set atomically and PG cannot
Forks settled:
- no snapshot format — the compacted log is the snapshot, existing grammar,
so no new encoder or decoder and the dump reuses wo_wal_append_insert
- one source, not two
- volume-only trigger, as a ratio against the LAST compaction's measured
output (the denominator is known exactly; estimating the live set would
mean estimating Text) with an absolute floor. NO TIMER — PG's exists to
bound loss from unflushed buffers and we have none; an idle log does not
grow. Copying the mechanism without the reason was the trap
- stop-the-world, with the pause measured against a stated budget rather
than assumed acceptable; alternatives are bought against a number
- compaction may run ONLY where nothing is staged (right after a barrier),
or a staged record lands in a file about to be replaced. Normative
Recorded before it can be found late: compaction invalidates every WAL
offset iteration 2's `resident: keys` stores, so the compactor rebuilds the
offset map as it writes. Nothing breaks today because that storage half is
unimplemented — it would break later, looking like corruption.
Also corrected exploration/postgresql/buffer-and-checkpoint.md, which was
wrong on two counts: PG does NOT update its control file by rename (in-place
full-block write + CRC32C), and its checkpoint sketch assumes writeonce has
segment files, which it does not and deliberately will not.
Grounding measured on master: seed 20000 leaves a 986614-byte log; 20000
updates take it to 2590262 bytes with the SAME live rows, and boot+verify on
that store is 155ms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
databasev2 4 part A, task 6. Mostly documentation, plus one real fix the
full battery caught.
THE FIX. The drain held EVERY DB reply until the barrier — including
reads, which stage nothing and have no stake in durability. That parked
readers behind an fsync for no reason: durable.sN.mixread.p99 rose from
~1043us to 4057us. Only a statement that actually staged a record now has
its reply held. Caught by the gate, not by review.
THE TRADE, recorded rather than smoothed over. What remains is inherent: a
barrier blocks the owner shard LONGER (more records per fsync) though LESS
OFTEN, so anything queued behind one waits. Three full runs of the same
build gave durable.sN.mixread.p99 of 1043 / 2318 / 4147us and wmix.p99 of
8758 / 20000us — a 2-4x spread with the box near idle. So part A buys ~3x
write throughput at the cost of a longer, noisier tail on the owner shard,
and that is the strongest argument for part B (submit and keep serving).
- durable.sN.*.p99us tolerance widened to 100% WITH the reason in the
code: a 2-4x-variable tail gated at 50% gates the disk, not the engine.
The floor is the real guard and is not slack — mixread's (4172us) came
within 25us of tripping on the worst run. Baseline refreshed; a fresh
full run then passed 106 checks 0 failures
EXIT STATUS MOVED 3 -> 74 (sysexits EX_IOERR). 3 and 4 are already used by
SAMPLES for their own meanings — db-bench's own `verify` exits 3 on a
checksum mismatch, and it is the gate that exercises durability, so a
durability abort exiting 3 would have been indistinguishable from the
mismatch it should help diagnose. The low range belongs to programs.
Docs:
- story: progress, the payoff measured two ways, the cost side, criteria
split met/outstanding, and a "part B — its premise changed" section:
it was justified by "close the 66x gap", but that gap is two problems
and only the concurrent one was a batching problem
- board: standup entry in the six-question shape; both databasev2 4 rows
rewritten. They had said "close the 66x gap" — recorded as MIS-STATED
rather than quietly renumbered
- 00-wob-format.md and 04-db-binding.md: the normative failure contract
("a failed WAL commit traps WO_T_IO after un-applying the row") was
false; corrected, along with the tick-scoped group commit that never
happened
- database/src/CODE-LOGIC.md: where the barrier runs and why there, why
replies are held, why the inline path is asymmetric, the one failure
rule, and how to measure it
- db-bench README: the wmix mode, the env knobs, and the tmpfs warning
Battery: wovm-test 36 suites 0 fail, woc-test, oop-e2e 119/0,
db-bench 106/0, linkcheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan for the approved spec. Code-free per the repo convention
(docs/plan/discarded.md:54); the executor writes the code.
- T1 a failed barrier is detected and fatal — one entry point that names
the operation, errno, WAL path and batch size, then exits. The abort
path itself stays unexercised and the task says so rather than buying
coverage with a fault-injection switch
- T2 the barrier moves to the drain point and replies are held; the
request path stops committing per append. Riskiest task, and its risk
is one place: the crash legs. Plan says STOP if they fail, do not
adjust the test
- T3 the inline path takes the same fatal rule but keeps its own barrier,
with a comment explaining the asymmetry so the next reader does not
"fix" it. Looks like a no-op; without it the two paths disagree, which
is the unevenness the spec exists to remove
- T4 prove batches actually form BEFORE measuring the payoff — otherwise
a win gets attributed to the wrong cause. Also records peak staged
bytes, settling the no-cap decision with a number
- T5 measure, gate, write it down. If the payoff is absent, say so and
stop: part B must not start on an unproven premise
- T6 closeout, including the error catalogue — WO_T_IO leaving the write
path is language-visible and must be written down
Spec corrected while planning: it pointed at durable.s1.seed as the
payoff. Wrong, structurally — worker shards hold no WAL, so a queue only
exists when other shards write, and a serial writer has nothing to batch
with. The real target is durable.sN.mixwrite: 480 ops/s at p99 5888us
against s1's 1023 at p99 664, so adding shards currently makes durable
writing WORSE. That inversion is a better argument for the iteration than
the one the story recorded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brainstormed 2026-08-28. The iteration is split: part A batches, part B
(io_uring submission) is deferred until A's measurement says whether the
blocking boundary still dominates.
The story's premise needed correcting first:
- it says "replace fsync-per-commit with io_uring group-commit", but the
engine commits per STATEMENT — db.c calls wo_wal_commit right after
every append, all six sites, so each row change is one pwrite + one
fdatasync
- so two independent wins were being carried as one, and only the second
needs io_uring. The staging buffer already holds any number of records;
today it never holds more than one. Part A is mostly deleting calls
- iteration 22's numbers say A is where the payoff is: durable writes
4460 ops/s, mixwrite 1023 ops/s p99 664us, against 1.28M ops/s reads
Forks settled:
- batch boundary is QUEUE-DRAIN, not the tick this story had recorded: a
tick adds latency to a lone writer, taxing an idle system to serve a
busy one. Queue-drain self-tunes and needs no knob
- shard 0 holds each reply envelope instead of sending it, commits once
when the queue empties, then releases all — so a writer is acked after
the barrier carrying ITS record, which today is true only because
every batch has one member
- a failure between "RAM mutated" and "record durable" is a FATAL,
diagnosed abort. This replaces uneven behaviour that already exists:
insert rolls back, update and delete do not and say so in a comment
("RAM ahead of disk"). Batching would have multiplied that
- consequence stated, not slipped in: WO_T_IO leaves the write path
- no batch cap initially; peak staged bytes is measured so the question
is settled by a number
One gap disclosed rather than hidden: forcing a real fdatasync failure
needs mount privileges, so the unit test proves the error is DETECTED and
the abort itself stays covered by inspection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
