- idempotency built, reviewed, then reverted WHOLE to the tag
archive/porch-idempotency. Not a design failure: it passed its gates.
It provokes a C-runtime SIGSEGV in wo_arena_alloc/wo_str_new under
concurrent call()-parked callers
- the evidence for that attribution: over ten gate runs every failure
was an idempotency leg and none was the limiter's, which drives the
same pool through the same call/park machinery. The begin arm has 5x
the allocation sites inside receive and moves a whole Req plus a
Handler through the mailbox
- before the split the suite reported 0 to 6 failures run to run; after
it, five consecutive runs at 56 checks, 0 failures
- PoolMsg loses digest/req/handler, and NullHandler/dummy_req/fresh_req
go with them — every rate-limit count used to allocate a throwaway
Req it never read
- IdempotencyKey is KEPT and commented: the schema is settled and the
digest-as-column decision cost a review round to get right
- the limiter's saturation 503 has no leg of its own now (§19 drove
Idempotent). Stated in the README rather than papered over — a
deterministic leg needs a slow actor, and only the reverted arm was
- new: porch 9 (idempotency, on hold) and language 41 (the arena crash,
with the reproduction harness and the evidence that localises it)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 79e6da4465133dc555e913c960d544ef1c7bedd8)
- story 02: `status: done`, `review_pending` (forks 1–7 auto-approved for
autonomy); progress rows 6a ✅, 6b ➡ databasev2 5 Phase A, 7 `a310496`;
5c/5d rows cite the `dev` hashes (the pre-merge ones were unreachable);
task 6a's Given/When/Then met; Info records the seven forks (sentinel over
`:memory:`, its rules, the refusal contract, startup-only, the budget
leaves for 5, library-owned tables bind consumers, the v8 table bit);
History keeps the first cut that refused every class-bearing program
- database/src/CODE-LOGIC.md: "Startup refusal + WO_EPHEMERAL" — contract,
hatch, table bit, measured blast radius, deferred items, proof; the
dispatcher paragraph no longer says a failed commit un-applies the row
(fatal since databasev2 4 part A; WO_T_IO unreachable from a write path)
- residency spec + plan: task 6 items annotated with the 2026-09-09
decisions; the byte budget marked moved to databasev2 5
- README, seven example READMEs and four guides carry the one-line rule
(durable default refuses without WO_DATA; WO_EPHEMERAL=1; durable:
false); shop's RAM-only command sets the sentinel
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 2c3531998124042fe736388e8b926abda3841194)
- five stories status: done; 00-story records the one-run landing
- spec History: three implementation amendments (Signal record not
scalar, caller-owned stdio fds, handler-latch instead of signalfd)
- board NEXT PLAN entry with measured findings (zero transport code
added; the tty-across-the-socket handover proven; the double-raw
refusal restoring the terminal — the "bug" that was the design
working); section rows flipped; graph nodes green
- CODE-LOGIC.md: the runtime-v2 section
- full belt quoted on the board: suites 0 fail both flavors (test_proc
193/0, test_term 60/0), woc 557/0, subprocess 12/0, site 23/0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit bc1b4f070693eb755ad6a9fd0c853fb3e2bda347)
- spec 2026-09-01-runtime-v2-design.md: the one principle (PULL — a
child is fds, the net verbs drive them; runtime-v2 adds acquisition
verbs, never transport), the full surface (ids 97+: spawn/spawn_pty/
wait_dl/signal/resize, signal.on delivering the sig number, term.raw/
restore with runtime-guaranteed restore, send_fd/recv_fd/connect_unix),
actor-owned lifecycle, mechanics notes, refusals by name
- push transport rejected with reasons recorded (mailbox-cap collision,
new delivery machinery); death-notice verb refused (a two-line fiber
composes wait_dl)
- five stories flip readiness: ready; fork sections rewritten as settled
- graph section 6 remapped: pull broke the 1->2->3 chain — only 1->2
remains; 3, 4, 5 and the VTE grid startable alone today
- board section + registry follow; linkcheck 0 broken
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit d313cdeebbe53c83b83f31f4480631568d9d0743)
- records every decision made without stopping to ask, with what each
costs if wrong, since the SDD workspace is deleted on completion
- R9 is marked WRONG and overturned by R14: WO-E222 fires on the class
Pool, not on multi, so an actor can hold slots: multi PoolSlot. My
ruling shipped a README prescribing a permanent 1-actor pool
- R6 records that my own brief caused a security bug: trust_proxy with
an absent XFF collapsed every client onto one shared bucket
- R15 parks the one residual: pool_slots/pool_of have zero call sites,
so real N-actor sharding is compile-proven but gate-unproven
- measured the gate over 10 runs: it is NOT stably green. Most runs
fail idempotent-stop; one lost 6 checks with 000 status codes
- traces the flake to the C-runtime hang/segfault, now localised by gdb
to wo_arena_alloc / wo_str_new / vm_run
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a919ab104ce44754949d364e35c88b6964b81fee)
- WO-E226: call's reply must be a copyable scalar, and every receive
program-wide must declare the same return type. Verified by fixture:
"call's reply type `Out` is not a copyable scalar"
- the spec had the actor return the response object, which cannot cross
the mailbox. Corrected: the actor stores the response and returns an
outcome code; the middleware reads the row and builds the Resp
- owner and duplicate now read the SAME durable row, so byte-identical
replay is structural rather than careful copying
- blocking, exactly-once execution and the mailbox queue are unchanged
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 77e06c1690b92d456a9bc53503695fdaa2b4b44e)
- one message class with a kind discriminator, not a receive per
message type: an actor handle is typed to one message class, so a
second receive compiles but is unreachable. chat/main.wo is the
precedent. Verified by fixture before amending
- Task 4's Files list omitted keypool.wo, which its step 4 edits
- clarified that the delete-then-insert ban targets using that pair as
an UPDATE; pruning an expired row is a plain delete and is required
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a96ebe20e92e2dc6dfecc59a955d4c6e75d74689)
- the spec's blocking design was unimplementable: call's reply IS the
return value of receive, so an actor cannot hold a waiter. Holding
means never returning, and an actor that never returns cannot process
the completion it waits for — deadlock
- corrected shape: the actor RUNS the handler inside its own receive, so
a duplicate waits in the mailbox and is served after the owner. The
queue blocking needs is the mailbox; nothing is held
- verified before adopting it, not after: an actor can receive a message
carrying an interface-typed value and invoke it, so the route's
Handler passes through the mailbox
- spec History records the reasoning error — "the primitives landed" was
taken as "blocking needs no new surface", which does not follow
- plan: 5 tasks. Counting and replay live in one new keypool.wo; both
middlewares become thin key-choosers, so porch 2 and 3 inherit one
serialization convention instead of re-implementing it
- self-review added two legs it was missing: exact counting under real
concurrency (the criterion the pool exists for), and pruning an
elapsed limiter row rather than resetting it, which otherwise leaks a
row per IP ever seen
- plan is code-free per house convention; the writing-plans skill wants
code blocks and the project rule overrides it
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f079455755a189a86bb12e07cc11549ed7a78b91)
- supersedes Phases B and C as built: a store-after-completion
middleware cannot satisfy three of the story's seven criteria
- in-flight collision is undetectable (the row is written after the
handler ran, so concurrent duplicates both miss and both execute)
- the 10s in-flight heuristic is inverted: created_at is stamped at
store time, so it fires on legitimate fast replays and never on a
genuinely concurrent request
- "reused key, different body is refused" is unreachable while the
digest is folded into the key — nothing looks the bare key up
- design: sharded actor pool serializes per key, @table persists;
actors own volatile state, tables own durability. Inherited by
porch 2 and 3
- limiter joins the pool for exact counting, writes through instead of
delete+insert, keys on net.peer unless trust_proxy is declared, and
uses monotonic ticks for arithmetic but wall clock for the header
- idempotency blocks rather than answering 409: call parks the
duplicate until the owner reports. Digest becomes a column
- saturation fails closed with 503 for both: saturating the pool must
not become the limiter bypass
- records that the story's "time.after is still reserved" is stale;
spawn/send/call/monitor/time.after all landed
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit fc09e94373db65837ff5eb620fec67bab02c1931)
- brainstorm settled: declarative and automatic at boot; v1 verbs are
add and delete only; data/seed migrations deferred to v2
- added fields zero-fill by kind: the grammar has no field-default
syntax and v1 refuses to grow compiler surface for it
- same-kind delete+add refuses as a disguised rename; retype and
vanished classes refuse by name
- schema lives in the log itself: WO_WAL_SCHEMA head record, written by
fresh-log open and compaction; name-keyed diff also closes the
silent cid-renumbering hole
- story is iteration 12, board row added, db2-migrate prefix claimed
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 072e007b144ff6689b6ff920ea10665900c1a2ef)
- flattened row image is WO_WAL_UPDATE, not WO_WAL_INSERT: the row's
original INSERT is already in a live log, so a second one for the same
id is a duplicate replay refuses as corruption. INSERT is right only
for compaction, which builds a fresh log
- remove WO_CKPT_MAX_GARBAGE: with the absolute term at 64 MiB, garbage
large enough to reach a 256 MiB ceiling has already tripped it, so the
branch was unreachable. Postgres needs both constants because it
thresholds on tuples with its pair at opposite ends; this thresholds
on bytes, where one constant does both jobs
- test_delta_chain_flattens_at_k: chain depth stays <= WO_DELTA_MAX_HOPS
across 2K+2 updates, and a reset is observed
- test_delta_chain_flatten_replays: a flattened chain replays correctly
- test_keys_resident_indexed_across_flatten: a delta on an indexed
column composes with flattening, checked at every step across the
bound and after restart. Found no product defect
- test_should_compact_absolute_and_ceiling: pins the absolute term, the
boundary just under it, and the small-log case the ratio still governs
- test_wal 5700 pass / 0 fail; wovm-test and woc-test green
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f93b5d9db753305c297e868d977670e6d703684c)
- fixes a limitation iteration 2 shipped: compaction was supposed to
bound chain length, but wo_wal_should_compact triggers on a whole-log
byte ratio and cannot see one hot row's chain
- tier 1, flatten on update: the update path ALREADY folds the row for
index maintenance and the fold already walks hop by hop, so it reports
depth for free. Past a fixed K it writes a full row instead of a
delta. Read <= K+1 reads, replay O(K^2) per row. No format change, no
per-row RAM, no new trigger
- tier 2: our compaction policy has a proportional term and a
SUPPRESSOR misleadingly called a floor; postgres's floor TRIGGERS on
small absolute garbage. Add that term and a ceiling
- design read from .dev/reference/postgresql, not recalled:
heap_page_prune_opt gates on an O(1) on-page hint then page fullness
against Max(fillfactor, BLCKSZ/10); autovacuum uses base + scale *
reltuples clamped by a max (50, 0.2, 1e8). Neither thresholds on
new-bytes-versus-old-bytes
- K deliberately does NOT scale with table size: postgres scales a
table-level aggregate with proportional harm, ours is per-row with
additive cost, so scaling up would make big databases boot worst
- the story says plainly it should NOT be next: task 7 has still never
measured whether resident: keys beats the kernel's own paging
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f667cad2cfbe187b5973440ab1af015b2df288f8)
- the plan placed the fold near wo_wal_read_row_at at "line ~600";
it is at line 794
- every other citation verified: wal.h:46, table.c:452/487,
db.c:88/289, wal.c:738, wal.c:861
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c6cd48679e21e510696391b340c735aa0ad098f9)
- 6 tasks: the record kind, the fold, updates + indexes, the request
path and group commit, replay + compaction, then lifting the loader
refusal and proving it end to end
- the fold is written ONCE and called from three places; the tests are
arranged to prove each caller separately, and the plan says that
wanting a second fold "just for this caller" means the design failed
- Task 2 includes a same-field ordering test specifically, because a
fold walking the chain backwards the wrong way returns plausible data
and is otherwise invisible
- Task 6 step 1 audits the db.c request arms BEFORE lifting anything —
they were never audited for keys-residency the way the inline path
was, and the last audit of that kind found delete corrupting memory
- self-review found the spec's crash criterion had no task: added a step
that commits a delta, skips the re-point, and replays, which is the
state a crash between barrier and flush leaves behind
- every symbol the plan names verified to exist in database/src
- code-free per house convention; the writing-plans skill wants code
blocks and the project rule overrides it
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit abb8fc9454bdca9d8d703c8cfeb224f89a93d14d)
- updates append a DELTA (class, id, field, value, back-pointer), not a
full row. The workload decides it: a product catalogue changes one
narrow field of a wide row on every order, so a full-row append would
rewrite every field to move one integer on a shop's hottest path
- the back-pointer keeps the id map at one slot per row, which is the
mode's whole premise; a map growing per update would defeat it
- ONE fold function, three callers (read, replay, compaction). Three
implementations of one rule is how they drift, and a fold that differs
between reading and replaying is a database that changes its mind at
boot. Named as the design's principal risk
- indexed columns MAY change: price is exactly what a catalogue indexes,
so forbidding it would be a restriction users meet immediately
- no chain cap, deliberately. Compaction already rewrites live rows, so
every checkpoint resets every chain, and deltas grow the log which
pulls the next checkpoint forward — the workload that lengthens chains
triggers the fold that shortens them
- the risk that accepts: one hot SKU under an otherwise quiet write
rate. Task 7's benchmark must include it
- supersedes the 2026-08-26 spec's one-line full-row Update sketch,
marked in place rather than left as a second design in the tree
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c9a88b05c4c62a5df13253686c990aaccc3f9cbb)
The branch was 17 ahead / 25 behind with 11 conflicting files, and drifting
further: db.c had been rewritten twice on master since (group commit, then
compaction). Resolved rather than rebased so both histories stay legible.
Conflicts, and how each was settled:
- db.c: BOTH semantics kept. Master's fatal path and compaction check now sit
behind the branch's `table_is_durable` predicate, in all three inline arms —
a volatile table reaches neither the barrier nor the compaction check
- db-bench sample: every mode from both sides (growth, growth-verify, randread,
replayseed, wmix) and ONE `boot` mode, which both sides had added
independently
- db-bench.py: all six legs kept. Both sides had also grown the same
WAL-size helper under different names; collapsed into one
- perf-targets: the branch's §5 (RAM ceiling) then master's §6/§7 — master's
numbering had already assumed a §5 it did not have
- story frontmatter: master's `status` (the landing truth) plus the branch's
`readiness` axis. 03 would have read `done` + `refine`, which is a
contradiction — it was brainstormed and landed on master, so `ready`
- board: both standup blocks newest-first; master's chain rows (a superset);
the branch's databasev2 1-2 rows with master's 3-4. Fixed a stray `|` in
master's row 3
- baseline: master's, then REGENERATED from a full campaign — 143 metrics,
132 checks, 0 failures with both sides' legs present
TWO HALF-EXPOSED FEATURES FIXED, because the merge rule is that master gets
no feature that is honoured in name only:
- `resident: keys` PARSED, set a .wob flag, and did nothing: rows stayed fully
resident. A developer could declare a 120 GB table keys-resident, watch it
compile, and be OOM-killed. The loader now REFUSES it with a message naming
what to write instead, until tasks 5c/5d land. The compiler still parses it
and its AST golden still passes, so the grammar work stays tested
- `durable: false` was honoured ONLY on the inline path. wo_db_exec_req had no
guard at all, so a volatile table written from an actor on a worker shard
would still be logged — precisely porch's session-table case, and precisely
what iteration 2 exists to provide. All three request-path arms now carry the
same predicate. Found by reading the merged code, not by a test: the obvious
probe runs main() on the primary and therefore only exercises the inline path
Verified on the merged tree: wovm-test 0, woc-test 0, oop-e2e 122/0,
residency-accept 8/0, db-bench 132/0, linkcheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan for the approved spec. Code-free per the repo convention
(docs/plan/discarded.md:54); the executor writes the code.
- T1 wo_wal_compact: walk live rows via the bitmap, append one INSERT
each through the EXISTING append path, fsync, rename over the live log,
fsync the parent dir, reopen the descriptor. Test asserts BOTH that the
log shrank AND that a replay reproduces the same rows/ids/values —
shorter alone is worthless, a truncating bug also passes that
- T2 a stale temp file is removed at open and never read. The test uses
PLAUSIBLE records, not garbage: garbage would be rejected anyway and
would prove nothing
- T3 the trigger as a PURE decision (used bytes, last compaction's
measured output, floor) so it is unit-testable without a store; env
knobs for floor and ratio, which is what makes the policy testable at
all. No timer, with the reason. The check is called only where nothing
is staged, asserted by a test that stages and expects deferral
- T4 kill -9 DURING compaction, extending the existing fork-based crash
battery. Asserts the PROPERTY — the store equals the pre- or the
post-compaction content, never a mixture, and every acked id survives.
Run repeatedly and state the count: it is a race, one green run proves
little
- T5 measure space reclaimed, boot before/after, and the stop-the-world
PAUSE against a stated budget. If the pause exceeds it, stop and report
— the alternatives are bought against that number, not before it
- T6 closeout, including the normative ordering rule in 04-db-binding.md
Constraints carried from the spec into every task:
- recovery must NOT change; a task editing the replay path should stop
- the dump must FLUSH PERIODICALLY. stage() grows the staging buffer by
doubling, so dumping a whole store through one buffer would hold the
entire store in RAM — the unbounded growth databasev2 1 identified as
how this engine dies
- a FAILED compaction is a missed optimisation, not a durability event,
so it must not take databasev2 4's fatal path
- gate tolerances must not be waived wholesale (part A's T4 made that
mistake), and the baseline is full-mode — writing a quick-mode baseline
over it is a regression part A also made
Deliberately NOT a task: rebuilding the `resident: keys` offset map. It
cannot be implemented against a feature that does not exist yet, so T6
records it as an obligation at the compactor and in the story instead of
a stub nobody can test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
databasev2 3, chain 6. Brainstormed 2026-08-28 after databasev2 4 part A
landed.
Design: compact the log by rewriting it as one record per live row into a
temp file, fsync, rename over the live WAL, fsync the parent dir, reopen.
Recovery is COMPLETELY UNCHANGED — boot still opens one file and replays
it — and the crash criterion ("the same store as if the checkpoint had
never started") is satisfied by rename, not by code we must get right.
Read .dev/reference/postgresql for this. The finding is that PG's design
is UNAVAILABLE to us, which is what makes the simpler option legitimate:
- PG never compacts its WAL; segments before the redo point are recycled
by rename or unlinked. Its records are page deltas, so a compacted redo
log is not a store — hence heap files, a control file, a redo pointer,
a second recovery source and a separate process
- ours are FULL ROW IMAGES (apply_record implements UPDATE as
remove-then-recreate), so a compacted log IS a complete store. That one
difference deletes all of the above from the design
- what IS worth porting is the ordering discipline: publish the new
"recovery starts here" atomically and LAST, so a crash falls back. PG
needs a start-of-checkpoint redo pointer plus an end-of-checkpoint
control file update; we get the same property from one rename, because
we can swap the whole data set atomically and PG cannot
Forks settled:
- no snapshot format — the compacted log is the snapshot, existing grammar,
so no new encoder or decoder and the dump reuses wo_wal_append_insert
- one source, not two
- volume-only trigger, as a ratio against the LAST compaction's measured
output (the denominator is known exactly; estimating the live set would
mean estimating Text) with an absolute floor. NO TIMER — PG's exists to
bound loss from unflushed buffers and we have none; an idle log does not
grow. Copying the mechanism without the reason was the trap
- stop-the-world, with the pause measured against a stated budget rather
than assumed acceptable; alternatives are bought against a number
- compaction may run ONLY where nothing is staged (right after a barrier),
or a staged record lands in a file about to be replaced. Normative
Recorded before it can be found late: compaction invalidates every WAL
offset iteration 2's `resident: keys` stores, so the compactor rebuilds the
offset map as it writes. Nothing breaks today because that storage half is
unimplemented — it would break later, looking like corruption.
Also corrected exploration/postgresql/buffer-and-checkpoint.md, which was
wrong on two counts: PG does NOT update its control file by rename (in-place
full-block write + CRC32C), and its checkpoint sketch assumes writeonce has
segment files, which it does not and deliberately will not.
Grounding measured on master: seed 20000 leaves a 986614-byte log; 20000
updates take it to 2590262 bytes with the SAME live rows, and boot+verify on
that store is 155ms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan for the approved spec. Code-free per the repo convention
(docs/plan/discarded.md:54); the executor writes the code.
- T1 a failed barrier is detected and fatal — one entry point that names
the operation, errno, WAL path and batch size, then exits. The abort
path itself stays unexercised and the task says so rather than buying
coverage with a fault-injection switch
- T2 the barrier moves to the drain point and replies are held; the
request path stops committing per append. Riskiest task, and its risk
is one place: the crash legs. Plan says STOP if they fail, do not
adjust the test
- T3 the inline path takes the same fatal rule but keeps its own barrier,
with a comment explaining the asymmetry so the next reader does not
"fix" it. Looks like a no-op; without it the two paths disagree, which
is the unevenness the spec exists to remove
- T4 prove batches actually form BEFORE measuring the payoff — otherwise
a win gets attributed to the wrong cause. Also records peak staged
bytes, settling the no-cap decision with a number
- T5 measure, gate, write it down. If the payoff is absent, say so and
stop: part B must not start on an unproven premise
- T6 closeout, including the error catalogue — WO_T_IO leaving the write
path is language-visible and must be written down
Spec corrected while planning: it pointed at durable.s1.seed as the
payoff. Wrong, structurally — worker shards hold no WAL, so a queue only
exists when other shards write, and a serial writer has nothing to batch
with. The real target is durable.sN.mixwrite: 480 ops/s at p99 5888us
against s1's 1023 at p99 664, so adding shards currently makes durable
writing WORSE. That inversion is a better argument for the iteration than
the one the story recorded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brainstormed 2026-08-28. The iteration is split: part A batches, part B
(io_uring submission) is deferred until A's measurement says whether the
blocking boundary still dominates.
The story's premise needed correcting first:
- it says "replace fsync-per-commit with io_uring group-commit", but the
engine commits per STATEMENT — db.c calls wo_wal_commit right after
every append, all six sites, so each row change is one pwrite + one
fdatasync
- so two independent wins were being carried as one, and only the second
needs io_uring. The staging buffer already holds any number of records;
today it never holds more than one. Part A is mostly deleting calls
- iteration 22's numbers say A is where the payoff is: durable writes
4460 ops/s, mixwrite 1023 ops/s p99 664us, against 1.28M ops/s reads
Forks settled:
- batch boundary is QUEUE-DRAIN, not the tick this story had recorded: a
tick adds latency to a lone writer, taxing an idle system to serve a
busy one. Queue-drain self-tunes and needs no knob
- shard 0 holds each reply envelope instead of sending it, commits once
when the queue empties, then releases all — so a writer is acked after
the barrier carrying ITS record, which today is true only because
every batch has one member
- a failure between "RAM mutated" and "record durable" is a FATAL,
diagnosed abort. This replaces uneven behaviour that already exists:
insert rolls back, update and delete do not and say so in a comment
("RAM ahead of disk"). Batching would have multiplied that
- consequence stated, not slipped in: WO_T_IO leaves the write path
- no batch cap initially; peak staged bytes is measured so the question
is settled by a number
One gap disclosed rather than hidden: forcing a real fdatasync failure
needs mount privileges, so the unit test proves the error is DETECTED and
the abort itself stays covered by inspection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Audit found 4 of 10 iterations citing it1 and 4 carrying stale claims the
measurement contradicts.
- 05: framing was contradicted, not merely incomplete. Its goal expected a
gradient to detect ("back-pressure before the cliff"); there is no cliff
— SIGKILL with swap off, exit 0 with swap on, and read latency STEPS
(1us -> 487us) rather than departing. Heading and goal rewritten; the
measurement makes the goal stronger, not weaker
- 05: budget must be bytes — 3.3x footprint spread — with headroom for
index doublings, else it fires during a rehash
- 05: new goal — eviction policy QUALITY is decisive, since getting the
resident set wrong costs 273x, not a few percent
- 06: its revival question now has a reference point. 273x is the KERNEL
SWAP path; `resident: keys` preads via page cache and must beat it. This
file revives only if 5c/5d lands near 273x rather than well below
- 04: write path is not where pressure bites (append ~1%, read 273x), so
the io_uring question that matters is iteration 2's deferred read-path
one, not group-commit
- 00-story: problem statement asserted the store "refuses the insert
rather than dying". Corrected in place — a banner above it was not
enough, a skimmer never reaches it
- residency spec: "swap thrash and the OOM killer" named exits that were
not measured; replaced with silence-or-a-corpse
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `randread N R` in the sample: fill N rows, read R across the WHOLE range
- Weyl order `i*2654435761 mod n` — no RNG in the language, none needed;
both legs read the SAME key order so residency is the only variable
- `randread` driver leg: control (256 MiB, does not bind) vs over-cap
(6 MiB + swap), sizes kept modest — quick resolves it in ~5s
- gates the RATIO, not the absolutes: over-cap reads/sec belongs to the
box's swap device, the factor between two runs belongs to the engine
- reads must all resolve (hits == R) or the leg fails; a collapse measured
over unresolved reads is noise
- 133 checks, 0 failures; gate bites on a doctored collapse_x
Measured — this closes the gap the swap leg left:
- resident 1 851 166 reads/sec, p50 0us p99 1us
- over-cap 6 771 reads/sec, p50 128us p99 487us
- 273x throughput, ~480x p99, all 20 000 reads resolving in both
- so the two access patterns sit ~270x apart under identical pressure:
append-mostly insert ~1%, random read 273x
- departure is a STEP not a curve (1us -> 487us, nothing between), which
is why p99_departure_decile finds no knee — there is none
- caveat recorded, NOT inherited: this is demand-paged anonymous memory
through swap (4 KiB/fault, no readahead). `resident: keys` preads via
the page cache — should be better, but databasev2 2 task 7 must measure
its own read path. New criterion added there
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `Wide` text-heavy reference shape beside Int-only `Item`
- `growth N int|text`: per-decile RSS read from own /proc/self/status
- `growth-verify`: survivor of a crash must be a contiguous intact prefix
- four footprint legs under a rootless cgroup v2 cap, swap on/off
- `ceiling` leg: die at the cap, then replay must come back intact
- footprint read as median-of-marginals; doublings a separate metric
- 121 checks, 0 failures; footprint gated ±10%, kill-timing ±100%
Measured, and it inverted two of the iteration's own predictions:
- footprint 96.5-100 B/row Int vs 320.6-324 B/row text = 3.3x, NOT the
"order of magnitude" three docs asserted
- table storage has NO checked ceiling: SIGKILL signal 9, not a catchable
WO_T_OOM. overcommit lets malloc succeed; kernel kills on page touch
- swap is NOT latency collapse: 900k rows 148s capped-with-swap vs 150s
uncapped. Append-mostly never re-touches cold pages
- ack-after-fsync survives an OOM kill: ~40k rows, no holes, no corruption
- iteration 2's budget dependency is REMOVED not satisfied — there is no
"swap onset" to derive a fraction from
- fix: subprocess returncode -9 was labelled a "checked refusal"; 137 is
the shell spelling of the same signal
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 5c step 1 of docs/superpowers/plans/2026-08-26-table-residency.md, as a
PURE REFACTOR: no storage change, no keys-table anywhere. Borrow is wo_row_ptr
plus a seam, every release is a no-op. Provable on its own before the storage
change it exists to enable.
DESIGN SETTLED BY READING THE STRUCTURES, and both answers make 5c smaller:
- the id hash needs NO new storage. `hvals` is already uint64 holding
slot+1 with 0 = empty (table.h), so offset+1 fits the same field, and the
interpretation is per-table because a table is wholly `all` or wholly
`keys`. No parallel map
- secondary indexes need NO change. `db_ibucket.ids` stores row IDS, not slot
indices, and table.c resolves them through the id hash. I had told the
developer these pointed at slab slots — that was wrong, and it is why this
is one shared accessor rather than 11 rewrites
- the real coupling is the unique shadow: idx_add_row and
row_apply_field_slot both FETCH the conflicting row and compare columns.
Both now borrow/release, so a keys-table's non-resident conflict will be
found rather than silently skipped — a unique check that only examines
resident rows is a correctness hole, not a limitation
The scratch lives on `db_table`, not on the stack and not per call. Per call
would allocate once per candidate inside a bucket loop, turning an O(1) probe
into an allocation storm; a stack buffer is unsafe because the loader bounds
field_cnt at 65535 (loader.c:189), so the worst case is ~512 KB. It is safe
per-table because the store is single-writer, and a `busy` flag is there to
catch a nested borrow rather than let it alias silently. Freed in
table_destroy.
Gates: all 18 runtime suites 0 fail under ASan+UBSan (test_table 856/0,
test_wal 3654/0), oop-e2e 119/0, residency 8/0, employee 8/0, db-actor 8/0.
And the pure-refactor proof the plan asked for: `db-bench --quick` 85/0, every
resident read/query/seed/write floor held — a refactor that moves a number is
not a refactor.
Remaining wo_row_ptr sites for 5d: 6 in table.c, 2 in db.c, 2 in wal.c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 5b of docs/superpowers/plans/2026-08-26-table-residency.md, whose Task 5
is now split 5a-5d (plan updated in this commit).
- the offset twin of wo_row_read (table.c:721): same out-gate contract —
every value handed back is a FRESH VM allocation — but resolved from a file
position instead of the id hash
- fits entirely in wal.c because everything it needs was already public:
scan_record and dec_val are local, and wo_val_decode_vm / wo_db_val_free
are exported at table.h:183-187. Two decode stages, since the record and
the VM speak different dialects: dec_val -> engine slots -> VM copies, with
the engine slots freed as scratch on every path
- ZERO storage change. Nothing calls it yet; that is the point of separating
it from 5c, so the read path can be proven before the slabs are touched
- refuses rather than guessing, each case distinguishable: no intact record
at the offset, a malformed header, a decode failure, trailing bytes, and a
REMOVE tombstone. That last one matters most — handing a tombstone back as
a row would read a deleted row as live
Tested by deep field comparison, not by "it parsed": 24 rows with a nil Text
every third row, each read back BY OFFSET and compared field by field,
including the string bytes. Plus all three refusal paths — tombstone, a
mid-record offset (the silent-wrong-row failure this guards), and past the
intact prefix.
The free-on-every-path claim is VERIFIED, not assumed: removing the free
produced 3 LeakSanitizer reports; restoring it returns to 0. Worth doing
because "ASan is clean" only means something if the harness would have
complained.
PLAN SPLIT: Task 5's storage half was written as if it were plumbing. Measured
instead: wo_row_ptr returns a db_row* into a slab with 11 call sites, table.c
has 37 slab references, db.c:105-181 scans slabs directly, enc_val serialises
FROM the slab, and no operation exists that drops a payload while keeping
index entries. Note this is the OPPOSITE half from the earlier retraction —
the record FORMAT needed nothing, the record STORAGE genuinely is deep. 5c
(id->offset map + drop-payload-keep-index) and 5d (rewiring the call sites,
scans, @unique/FK across the boundary) get their own write-ups.
Gates: test_wal 3654/0 (was 3428), all 18 runtime suites 0 fail under
ASan+UBSan, oop-e2e 119/0, residency 8/0, employee 8/0, db-actor 8/0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- found during the pre-execution review of the plan, before any code
- the claim was wrong in both spec and plan: table.c's db_val_encode builds
the IN-MEMORY slot; the FILE record is a separate encoding in wal.c and has
been flat since iteration 9. enc_val inlines every kind recursively with no
pointer anywhere; dec_val reads it back; a record is
`WO_WAL_INSERT | class_id | id | <value per field>` in the
len|crc|payload|mark frame; scan_record already preads and CRC-verifies a
record at an arbitrary offset
- so the row encoding needs NO change, and Task 5 (a "self-contained,
offset-based" rewrite billed as the iteration's substantive engineering) is
DELETED, not reduced. 8 tasks -> 7, and the highest-risk task is gone
- the real difficulty is where the spec never looked: wo_wal_append_insert
stages into a 1 MiB buffer, so a record's final offset is unknown until
flush. Threading an accurate offset back through a buffered writer —
correct across partial flush, failed commit and torn tail — is now Task 5's
first two steps, with a unit test that straddles a buffer boundary and a
case asserting no offset is published for a record that never reached disk
- dependent claims corrected: the mmap alternative's premise, the read-path
bullet (now names scan_record/dec_val), and the self-review coverage table,
which records the retraction rather than quietly dropping the row
- root cause worth noting: reading one layer and inferring another. Second
time this iteration — the first was assuming WO_HEAP_MB bounded table
storage when it bounds the VM arena
- no code written yet; linkcheck 0 broken / 0 anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- 8 tasks, 71 steps, from the 2026-08-26 table-residency spec
- deliberately contains NO code: discarded.md records "raw code in plan
documents" as rejected, and all six preceding plans have zero fences.
Stated in the header so it does not read as an omission. Every step
instead names the exact file and line region plus the required behaviour
- task order is by testable deliverable, not by layer:
1 grammar + defaults (no existing golden may move)
2 the cross-table check — a durable ref into a volatile table is refused
3 .wob v7: descriptor carries both properties, loader refuses the
meaningless combination so it never reaches the engine
4 durable:false skips the WAL at the three existing choke points in db.c;
replay refuses on mismatch rather than resurrecting rows
5 self-contained offset-based records — the one real rewrite, since
table.c returns a malloc'd address as the slot word today
6 resident:keys read path: id->offset map, pread, sequential scan;
@unique and FK-restrict across the boundary are the correctness core
7 the two runtime refusals — durable with no WO_DATA (silent data loss
today), and the resident-footprint budget
8 measure, baseline, crash battery, docs, closeout
- self-review table maps every spec section to a task. Two gaps found and
closed: the escape hatch for an intentionally ephemeral run (a refusal
with no way forward is worse than the loss it replaces), and persisting
the offset map in databasev2 3's snapshot
- linkcheck 0 broken / 0 anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `resident: all | keys` replaces `resident: all | index`. Two reasons beyond
taste: it kills the collision with the `index:` argument
(`@table(index: [customer], resident: index)` read badly), and it puts both
values on ONE axis — each now answers "what row data stays resident",
where `all`/`index` mixed a quantity with a structure name
- accurate as well as clearer: what stays resident is the id->offset map, the
secondary indexes and the unique shadows — all key structures; row payloads
are exactly what leaves. `resident: none` was rejected as overclaiming,
since the indexes very much are resident
- checked for collisions: neither `all` nor `keys` is a keyword or a builtin
(`key_at`/`val_at` exist, bare `keys` does not)
- the spec's wart note became a recorded decision; the rejected spelling is
kept quoted so the rationale still reads
- fixes a bug I introduced in the 2026-08-26 track move: all six moved
iterations carried a banner reading "Part of [Story — the database beyond
RAM]" whose link pointed at the LANGUAGE arc — correct target, lying text,
the exact failure mode the link audit warned about. Banners now point at
the databasev2 story, and the original "Part of" line says plainly which
track the iteration was authored in before the move
- linkcheck 0 broken / 0 anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- driving case: a 120 GB order table on a 32 GB host. Not a tuning problem;
no eviction policy fixes it. Developer accepted reconsidering the principle
- principle 7 rewritten: durability half UNCHANGED and unconditional
(WAL-logged, fsync before ack, CRC-dropped torn tail); residency half
demoted from law to per-table declaration. Old wording quoted in place so
the amendment is legible, with the reason: a doctrine a real workload
cannot satisfy gets ignored, and the failure it produced was an OOM kill
- spec: docs/superpowers/specs/2026-08-26-table-residency-design.md
One log-structured engine — the WAL already holds every row, so keep an
in-RAM id->offset map and pread rows back. No second engine, no user-space
row cache (the kernel page cache is the hot copy, which is already this
repo's stated position and why it avoids O_DIRECT)
- arithmetic that makes it work: 240M rows x 16 B of index = ~3.8 GB
resident in 32 GB. Indexes stay resident, rows do not. Buys ~2 orders of
magnitude, not infinity — stated plainly in the spec
- grammar: two optional keys, `durable: true|false` and `resident: all|index`,
both defaulting to today's behaviour, so all 28 existing @table
declarations compile untouched and no golden is reblessed
- rejected, with reasons recorded: mmap (rows are pointer-bearing —
table.c returns (uintptr_t)t as the slot word), buffer pool (the Rust-era
phase-12 design that died with that track), paged B-tree (stays rejected),
a three-valued enum, automatic spill, disk-backed-by-default
- self-review caught the budget defaulting to "none" while promising the ERP
developer a diagnostic instead of the OOM killer — contradiction fixed:
the budget defaults to a fraction of host memory, and its value comes from
databasev2 1's swap-onset measurement
- live docs that contradicted the amendment updated (subagent doctrine,
its guide, discarded.md's two rows, iteration 04's read claim, 07, 38);
dated specs/plans left as records. linkcheck 0 broken / 0 anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- README: shipped concurrency/HTTP/WebSockets sat in the roadmap as "not yet
available"; "no package manager" contradicted [deps]; the deps example
would not have compiled (the key IS the module name)
- runtime/README: leads with wovm, wo-rt.c demoted to a historical section;
dropped 2 nonexistent recipes, crates/rt, @gc refcounting, 13 suites -> 18
- employee + log-watcher READMEs claimed "does not compile"; both are gates
- error catalog: +10 emitted codes incl WO-E250, the only diagnostic the
shipped query surface raises; recorded why the sweep rotted
- language-surface: group-by parses, then the typechecker refuses it
- 00-code-review + 00-link-audit re-run; history kept, not rewritten
- 48 dead Rust-era exploration links de-linked rather than re-pointed (their
prose names the retired plan by number); successor map -> discarded.md
- 08-project-structure: compiler/plan/ never existed; corpus has 9 dirs, 5 empty
- releasing.md: dropped a --draft step the workflow never had
- new docs/00-doc-audit.md: findings + disposition, incl one row where the
audit was wrong and the doc it accused was right
- status folders removed: 34 stories flat, status only in frontmatter; 252
links recomputed from resolved paths; board/board-views/structure retaught
- story 24 -> in-progress, since frontmatter is now the only truth
- new iteration 38: fs mutation verbs + net.connect, the two capability
families no iteration owned
- new iteration 39: gofiber/fiber v3.5.0 parity study. The ledger called
CSRF/sessions unblocked by iteration 34's HMAC, but the runtime has no
source of randomness at all
- linkcheck skips .dev/.superpowers: 0 broken paths, 0 bad anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- runtime ids 91-95: net.read_dl/accept_dl/write_dl (per-call deadline,
nil/false = the EXPECTED timeout; ms<=0 = old behavior bit for bit),
net.listen_unix (unlink-before-bind, O_NONBLOCK on the listener —
probe-found: accept4's flag covers accepted sockets only), net.peer
- plane: one-op-per-park stays law — deadlines ride one per-shard
TIMEOUT tick (sentinel user_data) + post-CQE expiry sweep +
POLL_REMOVE tombstone; epoll's deadline scan grew the fd-park case;
fibers POOL instead of freeing mid-run (stale-CQE UAF); plain parks
zero park_deadline (no stale sleep deadlines)
- probe: all five seams verified on BOTH WO_IO backends (timeout
timing exact, peer round-trip, unix rebind)
- framework: parse_request grows first_ms/read_ms; serve_conn — the
keep-alive loop with deadlines where parked idle conns are LEGAL
(close-when-idle RETIRED); App.handle_conn exposes it; plain serve()
unchanged for simple apps
- web-app: app-owned accept_dl loop + ConnWorker actor per connection
(each builds its own App; cross-shard placement rides the DB actor);
WA_IDLE_MS knob; gate grows to 41 checks — two slow requests served
in PARALLEL, stalled client evicted at the idle deadline, slow-loris
torn at the read deadline (400)
- docs: story 35 -> done with banner; SQE/CQE design spec LANDED (was
the review doc); ledger rows (timeouts/unix/keep-alive/peer), graph
(NETSEAM cleared, KEEPAL done), builtin-surface rows, runtime
CODE-LOGIC section, board entry
- battery 13/13 fresh-built
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- one iteration by directive 2026-08-23: call() parks with typed reply
(envelope kinds 5/6 over the DB-RPC park), mailbox cap 1024 +
WO_T_ACTOR fail-fast, monitor(addr, msg) one-way, time.after
one-shot no-cancel
- story 34 resolved: C builtins sha1/sha256/hmac_sha256 over Bytes,
RFC vectors gated
- WS pure .wo: handler-owned upgrade (ws_accept + hijack sentinel),
frame codec over Bytes via 36's bitwise, two actors per connection
(sole-reader + sole-writer)
- chat sample: registry + room actors, python raw-RFC6455 gate —
cross-shard functional, 1k soak, SIGTERM drain, battery unchanged
- status PROPOSED — awaiting review before the plan
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- brings time.ticks builtin (id 84), bench sample + campaign driver
(scripts/db-bench.py), just db-bench/db-bench-quick recipes, first
baseline recorded, story 22 to done/, postgres study cards
- conflicts resolved: board in-progress table (iteration 36 +
framework rows kept, db-bench row now "22 landed"; dangling order
anchor repointed); story 36 moved back to in-progress/ (dir-rename
inference dragged it to done/ — 36 still awaits the manual pass)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- bench/baseline.json: 74 metrics from the first full campaign;
tolerances tuned by a two-run repeatability check (mix* 50%,
read/query 35%, rest 15% — rationale in _config)
- gate bites: --check mode; doctored copy fails, both real runs 74/0
- headline: durable seed 4.5k/s vs ram 297k/s (23's case); reads
O(table) at ~1.5k/s; mixread 1280 vs 21 ops/s single-vs-multi
(the arc's price); msgrate 13.4M vs 2.45M (mutex-inbox number)
- arc delta recorded in story 8; findings in sample README
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- worker DB builtins marshal to shard 0: requester-side slot encode
(VM heaps never read cross-shard), owner executes serialized in
adopt, reply unparks via new WO_PARK_INBOX park + envelope 3/4
- engine gains thread-agnostic slot entry points (insert_slots,
update_field_slot, val_encode/clone, wo_db_exec_req); traps and
messages byte-identical to the local path
- main.c: engine + replay boot BEFORE shards spawn; workers assert
rt.db/rt.wal NULL; busy shard adopts inbox once per slice
- latent stage-1 bug fixed: shared io_uring params static raced by
lazy worker init lost park wakes (~1/20 hangs); params per-vm,
short submit now fails loud
- new sample docs/examples/db-actor + just db-actor gate 8/0 (multi
x3, uring/epoll forced, single byte-exact, WAL replay pair);
ASan+TSan 6/6; full battery green
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- five-property map in marker doc: atomicity/durability/recovery
proven or held (18); concurrency control = stage 3's property;
space reclamation RAM done (slot reuse), disk = new story 32
- story 08: three stage-3 criteria (one commit per write RPC +
ack-after-owner-fsync, workers WAL-free + replay-before-serve,
no torn reads under TSan corpus); arc plan stage 3 carries them
- refine/32-wal-checkpoint.md: snapshot + truncate, bounded replay,
crash-during-checkpoint safe; four forks; after 23
- chain now stage 3 -> 22 -> 31 -> 24 -> 23 -> 32 in all 10 docs;
boards + seq bumps synced
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- 08 landed in discarded/ by mis-drop, swept into prior commit with
two dead links; corrected to stories/.../in-progress/ per intent
- 11 joins it (one arc, active slice)
- board doctrine: active stories bucket named; all links re-verified
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- io_uring is a must; epoll approach discarded (developer decision)
- plan superseded by shard-fiber-arc plan of record; banner + row in
plan/discarded.md; file kept as idea reference
- three live pointers repointed: status language-track row 8,
principles enforced-by, story 08 note
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- code-verified ready: arc stages 1+2 merged to master; stage 3
concrete in plan (tasks 7-8); rt.db set on primary only
(main.c), worker_late_init memsets rt — WO_T_DB hole real
- 22/23/24/31 stay in refine/: open forks, no bench harness,
no crypto builtins, chain-blocked
- links fixed both directions; board doctrine names hold/ bucket
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- iterations 18/25/26 marked hold in their spec + plan headers
- 25's story file removal committed; plan doc stays for resumption
- web-framework spec's relates-to flags 25 held
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
