- updates append a DELTA (class, id, field, value, back-pointer), not a
full row. The workload decides it: a product catalogue changes one
narrow field of a wide row on every order, so a full-row append would
rewrite every field to move one integer on a shop's hottest path
- the back-pointer keeps the id map at one slot per row, which is the
mode's whole premise; a map growing per update would defeat it
- ONE fold function, three callers (read, replay, compaction). Three
implementations of one rule is how they drift, and a fold that differs
between reading and replaying is a database that changes its mind at
boot. Named as the design's principal risk
- indexed columns MAY change: price is exactly what a catalogue indexes,
so forbidding it would be a restriction users meet immediately
- no chain cap, deliberately. Compaction already rewrites live rows, so
every checkpoint resets every chain, and deltas grow the log which
pulls the next checkpoint forward — the workload that lengthens chains
triggers the fold that shortens them
- the risk that accepts: one hot SKU under an otherwise quiet write
rate. Task 7's benchmark must include it
- supersedes the 2026-08-26 spec's one-line full-row Update sketch,
marked in place rather than left as a second design in the tree
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c9a88b05c4c62a5df13253686c990aaccc3f9cbb)
The branch was 17 ahead / 25 behind with 11 conflicting files, and drifting
further: db.c had been rewritten twice on master since (group commit, then
compaction). Resolved rather than rebased so both histories stay legible.
Conflicts, and how each was settled:
- db.c: BOTH semantics kept. Master's fatal path and compaction check now sit
behind the branch's `table_is_durable` predicate, in all three inline arms —
a volatile table reaches neither the barrier nor the compaction check
- db-bench sample: every mode from both sides (growth, growth-verify, randread,
replayseed, wmix) and ONE `boot` mode, which both sides had added
independently
- db-bench.py: all six legs kept. Both sides had also grown the same
WAL-size helper under different names; collapsed into one
- perf-targets: the branch's §5 (RAM ceiling) then master's §6/§7 — master's
numbering had already assumed a §5 it did not have
- story frontmatter: master's `status` (the landing truth) plus the branch's
`readiness` axis. 03 would have read `done` + `refine`, which is a
contradiction — it was brainstormed and landed on master, so `ready`
- board: both standup blocks newest-first; master's chain rows (a superset);
the branch's databasev2 1-2 rows with master's 3-4. Fixed a stray `|` in
master's row 3
- baseline: master's, then REGENERATED from a full campaign — 143 metrics,
132 checks, 0 failures with both sides' legs present
TWO HALF-EXPOSED FEATURES FIXED, because the merge rule is that master gets
no feature that is honoured in name only:
- `resident: keys` PARSED, set a .wob flag, and did nothing: rows stayed fully
resident. A developer could declare a 120 GB table keys-resident, watch it
compile, and be OOM-killed. The loader now REFUSES it with a message naming
what to write instead, until tasks 5c/5d land. The compiler still parses it
and its AST golden still passes, so the grammar work stays tested
- `durable: false` was honoured ONLY on the inline path. wo_db_exec_req had no
guard at all, so a volatile table written from an actor on a worker shard
would still be logged — precisely porch's session-table case, and precisely
what iteration 2 exists to provide. All three request-path arms now carry the
same predicate. Found by reading the merged code, not by a test: the obvious
probe runs main() on the primary and therefore only exercises the inline path
Verified on the merged tree: wovm-test 0, woc-test 0, oop-e2e 122/0,
residency-accept 8/0, db-bench 132/0, linkcheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan for the approved spec. Code-free per the repo convention
(docs/plan/discarded.md:54); the executor writes the code.
- T1 wo_wal_compact: walk live rows via the bitmap, append one INSERT
each through the EXISTING append path, fsync, rename over the live log,
fsync the parent dir, reopen the descriptor. Test asserts BOTH that the
log shrank AND that a replay reproduces the same rows/ids/values —
shorter alone is worthless, a truncating bug also passes that
- T2 a stale temp file is removed at open and never read. The test uses
PLAUSIBLE records, not garbage: garbage would be rejected anyway and
would prove nothing
- T3 the trigger as a PURE decision (used bytes, last compaction's
measured output, floor) so it is unit-testable without a store; env
knobs for floor and ratio, which is what makes the policy testable at
all. No timer, with the reason. The check is called only where nothing
is staged, asserted by a test that stages and expects deferral
- T4 kill -9 DURING compaction, extending the existing fork-based crash
battery. Asserts the PROPERTY — the store equals the pre- or the
post-compaction content, never a mixture, and every acked id survives.
Run repeatedly and state the count: it is a race, one green run proves
little
- T5 measure space reclaimed, boot before/after, and the stop-the-world
PAUSE against a stated budget. If the pause exceeds it, stop and report
— the alternatives are bought against that number, not before it
- T6 closeout, including the normative ordering rule in 04-db-binding.md
Constraints carried from the spec into every task:
- recovery must NOT change; a task editing the replay path should stop
- the dump must FLUSH PERIODICALLY. stage() grows the staging buffer by
doubling, so dumping a whole store through one buffer would hold the
entire store in RAM — the unbounded growth databasev2 1 identified as
how this engine dies
- a FAILED compaction is a missed optimisation, not a durability event,
so it must not take databasev2 4's fatal path
- gate tolerances must not be waived wholesale (part A's T4 made that
mistake), and the baseline is full-mode — writing a quick-mode baseline
over it is a regression part A also made
Deliberately NOT a task: rebuilding the `resident: keys` offset map. It
cannot be implemented against a feature that does not exist yet, so T6
records it as an obligation at the compactor and in the story instead of
a stub nobody can test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
databasev2 3, chain 6. Brainstormed 2026-08-28 after databasev2 4 part A
landed.
Design: compact the log by rewriting it as one record per live row into a
temp file, fsync, rename over the live WAL, fsync the parent dir, reopen.
Recovery is COMPLETELY UNCHANGED — boot still opens one file and replays
it — and the crash criterion ("the same store as if the checkpoint had
never started") is satisfied by rename, not by code we must get right.
Read .dev/reference/postgresql for this. The finding is that PG's design
is UNAVAILABLE to us, which is what makes the simpler option legitimate:
- PG never compacts its WAL; segments before the redo point are recycled
by rename or unlinked. Its records are page deltas, so a compacted redo
log is not a store — hence heap files, a control file, a redo pointer,
a second recovery source and a separate process
- ours are FULL ROW IMAGES (apply_record implements UPDATE as
remove-then-recreate), so a compacted log IS a complete store. That one
difference deletes all of the above from the design
- what IS worth porting is the ordering discipline: publish the new
"recovery starts here" atomically and LAST, so a crash falls back. PG
needs a start-of-checkpoint redo pointer plus an end-of-checkpoint
control file update; we get the same property from one rename, because
we can swap the whole data set atomically and PG cannot
Forks settled:
- no snapshot format — the compacted log is the snapshot, existing grammar,
so no new encoder or decoder and the dump reuses wo_wal_append_insert
- one source, not two
- volume-only trigger, as a ratio against the LAST compaction's measured
output (the denominator is known exactly; estimating the live set would
mean estimating Text) with an absolute floor. NO TIMER — PG's exists to
bound loss from unflushed buffers and we have none; an idle log does not
grow. Copying the mechanism without the reason was the trap
- stop-the-world, with the pause measured against a stated budget rather
than assumed acceptable; alternatives are bought against a number
- compaction may run ONLY where nothing is staged (right after a barrier),
or a staged record lands in a file about to be replaced. Normative
Recorded before it can be found late: compaction invalidates every WAL
offset iteration 2's `resident: keys` stores, so the compactor rebuilds the
offset map as it writes. Nothing breaks today because that storage half is
unimplemented — it would break later, looking like corruption.
Also corrected exploration/postgresql/buffer-and-checkpoint.md, which was
wrong on two counts: PG does NOT update its control file by rename (in-place
full-block write + CRC32C), and its checkpoint sketch assumes writeonce has
segment files, which it does not and deliberately will not.
Grounding measured on master: seed 20000 leaves a 986614-byte log; 20000
updates take it to 2590262 bytes with the SAME live rows, and boot+verify on
that store is 155ms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan for the approved spec. Code-free per the repo convention
(docs/plan/discarded.md:54); the executor writes the code.
- T1 a failed barrier is detected and fatal — one entry point that names
the operation, errno, WAL path and batch size, then exits. The abort
path itself stays unexercised and the task says so rather than buying
coverage with a fault-injection switch
- T2 the barrier moves to the drain point and replies are held; the
request path stops committing per append. Riskiest task, and its risk
is one place: the crash legs. Plan says STOP if they fail, do not
adjust the test
- T3 the inline path takes the same fatal rule but keeps its own barrier,
with a comment explaining the asymmetry so the next reader does not
"fix" it. Looks like a no-op; without it the two paths disagree, which
is the unevenness the spec exists to remove
- T4 prove batches actually form BEFORE measuring the payoff — otherwise
a win gets attributed to the wrong cause. Also records peak staged
bytes, settling the no-cap decision with a number
- T5 measure, gate, write it down. If the payoff is absent, say so and
stop: part B must not start on an unproven premise
- T6 closeout, including the error catalogue — WO_T_IO leaving the write
path is language-visible and must be written down
Spec corrected while planning: it pointed at durable.s1.seed as the
payoff. Wrong, structurally — worker shards hold no WAL, so a queue only
exists when other shards write, and a serial writer has nothing to batch
with. The real target is durable.sN.mixwrite: 480 ops/s at p99 5888us
against s1's 1023 at p99 664, so adding shards currently makes durable
writing WORSE. That inversion is a better argument for the iteration than
the one the story recorded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brainstormed 2026-08-28. The iteration is split: part A batches, part B
(io_uring submission) is deferred until A's measurement says whether the
blocking boundary still dominates.
The story's premise needed correcting first:
- it says "replace fsync-per-commit with io_uring group-commit", but the
engine commits per STATEMENT — db.c calls wo_wal_commit right after
every append, all six sites, so each row change is one pwrite + one
fdatasync
- so two independent wins were being carried as one, and only the second
needs io_uring. The staging buffer already holds any number of records;
today it never holds more than one. Part A is mostly deleting calls
- iteration 22's numbers say A is where the payoff is: durable writes
4460 ops/s, mixwrite 1023 ops/s p99 664us, against 1.28M ops/s reads
Forks settled:
- batch boundary is QUEUE-DRAIN, not the tick this story had recorded: a
tick adds latency to a lone writer, taxing an idle system to serve a
busy one. Queue-drain self-tunes and needs no knob
- shard 0 holds each reply envelope instead of sending it, commits once
when the queue empties, then releases all — so a writer is acked after
the barrier carrying ITS record, which today is true only because
every batch has one member
- a failure between "RAM mutated" and "record durable" is a FATAL,
diagnosed abort. This replaces uneven behaviour that already exists:
insert rolls back, update and delete do not and say so in a comment
("RAM ahead of disk"). Batching would have multiplied that
- consequence stated, not slipped in: WO_T_IO leaves the write path
- no batch cap initially; peak staged bytes is measured so the question
is settled by a number
One gap disclosed rather than hidden: forcing a real fdatasync failure
needs mount privileges, so the unit test proves the error is DETECTED and
the abort itself stays covered by inspection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Audit found 4 of 10 iterations citing it1 and 4 carrying stale claims the
measurement contradicts.
- 05: framing was contradicted, not merely incomplete. Its goal expected a
gradient to detect ("back-pressure before the cliff"); there is no cliff
— SIGKILL with swap off, exit 0 with swap on, and read latency STEPS
(1us -> 487us) rather than departing. Heading and goal rewritten; the
measurement makes the goal stronger, not weaker
- 05: budget must be bytes — 3.3x footprint spread — with headroom for
index doublings, else it fires during a rehash
- 05: new goal — eviction policy QUALITY is decisive, since getting the
resident set wrong costs 273x, not a few percent
- 06: its revival question now has a reference point. 273x is the KERNEL
SWAP path; `resident: keys` preads via page cache and must beat it. This
file revives only if 5c/5d lands near 273x rather than well below
- 04: write path is not where pressure bites (append ~1%, read 273x), so
the io_uring question that matters is iteration 2's deferred read-path
one, not group-commit
- 00-story: problem statement asserted the store "refuses the insert
rather than dying". Corrected in place — a banner above it was not
enough, a skimmer never reaches it
- residency spec: "swap thrash and the OOM killer" named exits that were
not measured; replaced with silence-or-a-corpse
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `randread N R` in the sample: fill N rows, read R across the WHOLE range
- Weyl order `i*2654435761 mod n` — no RNG in the language, none needed;
both legs read the SAME key order so residency is the only variable
- `randread` driver leg: control (256 MiB, does not bind) vs over-cap
(6 MiB + swap), sizes kept modest — quick resolves it in ~5s
- gates the RATIO, not the absolutes: over-cap reads/sec belongs to the
box's swap device, the factor between two runs belongs to the engine
- reads must all resolve (hits == R) or the leg fails; a collapse measured
over unresolved reads is noise
- 133 checks, 0 failures; gate bites on a doctored collapse_x
Measured — this closes the gap the swap leg left:
- resident 1 851 166 reads/sec, p50 0us p99 1us
- over-cap 6 771 reads/sec, p50 128us p99 487us
- 273x throughput, ~480x p99, all 20 000 reads resolving in both
- so the two access patterns sit ~270x apart under identical pressure:
append-mostly insert ~1%, random read 273x
- departure is a STEP not a curve (1us -> 487us, nothing between), which
is why p99_departure_decile finds no knee — there is none
- caveat recorded, NOT inherited: this is demand-paged anonymous memory
through swap (4 KiB/fault, no readahead). `resident: keys` preads via
the page cache — should be better, but databasev2 2 task 7 must measure
its own read path. New criterion added there
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `Wide` text-heavy reference shape beside Int-only `Item`
- `growth N int|text`: per-decile RSS read from own /proc/self/status
- `growth-verify`: survivor of a crash must be a contiguous intact prefix
- four footprint legs under a rootless cgroup v2 cap, swap on/off
- `ceiling` leg: die at the cap, then replay must come back intact
- footprint read as median-of-marginals; doublings a separate metric
- 121 checks, 0 failures; footprint gated ±10%, kill-timing ±100%
Measured, and it inverted two of the iteration's own predictions:
- footprint 96.5-100 B/row Int vs 320.6-324 B/row text = 3.3x, NOT the
"order of magnitude" three docs asserted
- table storage has NO checked ceiling: SIGKILL signal 9, not a catchable
WO_T_OOM. overcommit lets malloc succeed; kernel kills on page touch
- swap is NOT latency collapse: 900k rows 148s capped-with-swap vs 150s
uncapped. Append-mostly never re-touches cold pages
- ack-after-fsync survives an OOM kill: ~40k rows, no holes, no corruption
- iteration 2's budget dependency is REMOVED not satisfied — there is no
"swap onset" to derive a fraction from
- fix: subprocess returncode -9 was labelled a "checked refusal"; 137 is
the shell spelling of the same signal
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 5c step 1 of docs/superpowers/plans/2026-08-26-table-residency.md, as a
PURE REFACTOR: no storage change, no keys-table anywhere. Borrow is wo_row_ptr
plus a seam, every release is a no-op. Provable on its own before the storage
change it exists to enable.
DESIGN SETTLED BY READING THE STRUCTURES, and both answers make 5c smaller:
- the id hash needs NO new storage. `hvals` is already uint64 holding
slot+1 with 0 = empty (table.h), so offset+1 fits the same field, and the
interpretation is per-table because a table is wholly `all` or wholly
`keys`. No parallel map
- secondary indexes need NO change. `db_ibucket.ids` stores row IDS, not slot
indices, and table.c resolves them through the id hash. I had told the
developer these pointed at slab slots — that was wrong, and it is why this
is one shared accessor rather than 11 rewrites
- the real coupling is the unique shadow: idx_add_row and
row_apply_field_slot both FETCH the conflicting row and compare columns.
Both now borrow/release, so a keys-table's non-resident conflict will be
found rather than silently skipped — a unique check that only examines
resident rows is a correctness hole, not a limitation
The scratch lives on `db_table`, not on the stack and not per call. Per call
would allocate once per candidate inside a bucket loop, turning an O(1) probe
into an allocation storm; a stack buffer is unsafe because the loader bounds
field_cnt at 65535 (loader.c:189), so the worst case is ~512 KB. It is safe
per-table because the store is single-writer, and a `busy` flag is there to
catch a nested borrow rather than let it alias silently. Freed in
table_destroy.
Gates: all 18 runtime suites 0 fail under ASan+UBSan (test_table 856/0,
test_wal 3654/0), oop-e2e 119/0, residency 8/0, employee 8/0, db-actor 8/0.
And the pure-refactor proof the plan asked for: `db-bench --quick` 85/0, every
resident read/query/seed/write floor held — a refactor that moves a number is
not a refactor.
Remaining wo_row_ptr sites for 5d: 6 in table.c, 2 in db.c, 2 in wal.c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 5b of docs/superpowers/plans/2026-08-26-table-residency.md, whose Task 5
is now split 5a-5d (plan updated in this commit).
- the offset twin of wo_row_read (table.c:721): same out-gate contract —
every value handed back is a FRESH VM allocation — but resolved from a file
position instead of the id hash
- fits entirely in wal.c because everything it needs was already public:
scan_record and dec_val are local, and wo_val_decode_vm / wo_db_val_free
are exported at table.h:183-187. Two decode stages, since the record and
the VM speak different dialects: dec_val -> engine slots -> VM copies, with
the engine slots freed as scratch on every path
- ZERO storage change. Nothing calls it yet; that is the point of separating
it from 5c, so the read path can be proven before the slabs are touched
- refuses rather than guessing, each case distinguishable: no intact record
at the offset, a malformed header, a decode failure, trailing bytes, and a
REMOVE tombstone. That last one matters most — handing a tombstone back as
a row would read a deleted row as live
Tested by deep field comparison, not by "it parsed": 24 rows with a nil Text
every third row, each read back BY OFFSET and compared field by field,
including the string bytes. Plus all three refusal paths — tombstone, a
mid-record offset (the silent-wrong-row failure this guards), and past the
intact prefix.
The free-on-every-path claim is VERIFIED, not assumed: removing the free
produced 3 LeakSanitizer reports; restoring it returns to 0. Worth doing
because "ASan is clean" only means something if the harness would have
complained.
PLAN SPLIT: Task 5's storage half was written as if it were plumbing. Measured
instead: wo_row_ptr returns a db_row* into a slab with 11 call sites, table.c
has 37 slab references, db.c:105-181 scans slabs directly, enc_val serialises
FROM the slab, and no operation exists that drops a payload while keeping
index entries. Note this is the OPPOSITE half from the earlier retraction —
the record FORMAT needed nothing, the record STORAGE genuinely is deep. 5c
(id->offset map + drop-payload-keep-index) and 5d (rewiring the call sites,
scans, @unique/FK across the boundary) get their own write-ups.
Gates: test_wal 3654/0 (was 3428), all 18 runtime suites 0 fail under
ASan+UBSan, oop-e2e 119/0, residency 8/0, employee 8/0, db-actor 8/0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- found during the pre-execution review of the plan, before any code
- the claim was wrong in both spec and plan: table.c's db_val_encode builds
the IN-MEMORY slot; the FILE record is a separate encoding in wal.c and has
been flat since iteration 9. enc_val inlines every kind recursively with no
pointer anywhere; dec_val reads it back; a record is
`WO_WAL_INSERT | class_id | id | <value per field>` in the
len|crc|payload|mark frame; scan_record already preads and CRC-verifies a
record at an arbitrary offset
- so the row encoding needs NO change, and Task 5 (a "self-contained,
offset-based" rewrite billed as the iteration's substantive engineering) is
DELETED, not reduced. 8 tasks -> 7, and the highest-risk task is gone
- the real difficulty is where the spec never looked: wo_wal_append_insert
stages into a 1 MiB buffer, so a record's final offset is unknown until
flush. Threading an accurate offset back through a buffered writer —
correct across partial flush, failed commit and torn tail — is now Task 5's
first two steps, with a unit test that straddles a buffer boundary and a
case asserting no offset is published for a record that never reached disk
- dependent claims corrected: the mmap alternative's premise, the read-path
bullet (now names scan_record/dec_val), and the self-review coverage table,
which records the retraction rather than quietly dropping the row
- root cause worth noting: reading one layer and inferring another. Second
time this iteration — the first was assuming WO_HEAP_MB bounded table
storage when it bounds the VM arena
- no code written yet; linkcheck 0 broken / 0 anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- 8 tasks, 71 steps, from the 2026-08-26 table-residency spec
- deliberately contains NO code: discarded.md records "raw code in plan
documents" as rejected, and all six preceding plans have zero fences.
Stated in the header so it does not read as an omission. Every step
instead names the exact file and line region plus the required behaviour
- task order is by testable deliverable, not by layer:
1 grammar + defaults (no existing golden may move)
2 the cross-table check — a durable ref into a volatile table is refused
3 .wob v7: descriptor carries both properties, loader refuses the
meaningless combination so it never reaches the engine
4 durable:false skips the WAL at the three existing choke points in db.c;
replay refuses on mismatch rather than resurrecting rows
5 self-contained offset-based records — the one real rewrite, since
table.c returns a malloc'd address as the slot word today
6 resident:keys read path: id->offset map, pread, sequential scan;
@unique and FK-restrict across the boundary are the correctness core
7 the two runtime refusals — durable with no WO_DATA (silent data loss
today), and the resident-footprint budget
8 measure, baseline, crash battery, docs, closeout
- self-review table maps every spec section to a task. Two gaps found and
closed: the escape hatch for an intentionally ephemeral run (a refusal
with no way forward is worse than the loss it replaces), and persisting
the offset map in databasev2 3's snapshot
- linkcheck 0 broken / 0 anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `resident: all | keys` replaces `resident: all | index`. Two reasons beyond
taste: it kills the collision with the `index:` argument
(`@table(index: [customer], resident: index)` read badly), and it puts both
values on ONE axis — each now answers "what row data stays resident",
where `all`/`index` mixed a quantity with a structure name
- accurate as well as clearer: what stays resident is the id->offset map, the
secondary indexes and the unique shadows — all key structures; row payloads
are exactly what leaves. `resident: none` was rejected as overclaiming,
since the indexes very much are resident
- checked for collisions: neither `all` nor `keys` is a keyword or a builtin
(`key_at`/`val_at` exist, bare `keys` does not)
- the spec's wart note became a recorded decision; the rejected spelling is
kept quoted so the rationale still reads
- fixes a bug I introduced in the 2026-08-26 track move: all six moved
iterations carried a banner reading "Part of [Story — the database beyond
RAM]" whose link pointed at the LANGUAGE arc — correct target, lying text,
the exact failure mode the link audit warned about. Banners now point at
the databasev2 story, and the original "Part of" line says plainly which
track the iteration was authored in before the move
- linkcheck 0 broken / 0 anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- driving case: a 120 GB order table on a 32 GB host. Not a tuning problem;
no eviction policy fixes it. Developer accepted reconsidering the principle
- principle 7 rewritten: durability half UNCHANGED and unconditional
(WAL-logged, fsync before ack, CRC-dropped torn tail); residency half
demoted from law to per-table declaration. Old wording quoted in place so
the amendment is legible, with the reason: a doctrine a real workload
cannot satisfy gets ignored, and the failure it produced was an OOM kill
- spec: docs/superpowers/specs/2026-08-26-table-residency-design.md
One log-structured engine — the WAL already holds every row, so keep an
in-RAM id->offset map and pread rows back. No second engine, no user-space
row cache (the kernel page cache is the hot copy, which is already this
repo's stated position and why it avoids O_DIRECT)
- arithmetic that makes it work: 240M rows x 16 B of index = ~3.8 GB
resident in 32 GB. Indexes stay resident, rows do not. Buys ~2 orders of
magnitude, not infinity — stated plainly in the spec
- grammar: two optional keys, `durable: true|false` and `resident: all|index`,
both defaulting to today's behaviour, so all 28 existing @table
declarations compile untouched and no golden is reblessed
- rejected, with reasons recorded: mmap (rows are pointer-bearing —
table.c returns (uintptr_t)t as the slot word), buffer pool (the Rust-era
phase-12 design that died with that track), paged B-tree (stays rejected),
a three-valued enum, automatic spill, disk-backed-by-default
- self-review caught the budget defaulting to "none" while promising the ERP
developer a diagnostic instead of the OOM killer — contradiction fixed:
the budget defaults to a fraction of host memory, and its value comes from
databasev2 1's swap-onset measurement
- live docs that contradicted the amendment updated (subagent doctrine,
its guide, discarded.md's two rows, iteration 04's read claim, 07, 38);
dated specs/plans left as records. linkcheck 0 broken / 0 anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- README: shipped concurrency/HTTP/WebSockets sat in the roadmap as "not yet
available"; "no package manager" contradicted [deps]; the deps example
would not have compiled (the key IS the module name)
- runtime/README: leads with wovm, wo-rt.c demoted to a historical section;
dropped 2 nonexistent recipes, crates/rt, @gc refcounting, 13 suites -> 18
- employee + log-watcher READMEs claimed "does not compile"; both are gates
- error catalog: +10 emitted codes incl WO-E250, the only diagnostic the
shipped query surface raises; recorded why the sweep rotted
- language-surface: group-by parses, then the typechecker refuses it
- 00-code-review + 00-link-audit re-run; history kept, not rewritten
- 48 dead Rust-era exploration links de-linked rather than re-pointed (their
prose names the retired plan by number); successor map -> discarded.md
- 08-project-structure: compiler/plan/ never existed; corpus has 9 dirs, 5 empty
- releasing.md: dropped a --draft step the workflow never had
- new docs/00-doc-audit.md: findings + disposition, incl one row where the
audit was wrong and the doc it accused was right
- status folders removed: 34 stories flat, status only in frontmatter; 252
links recomputed from resolved paths; board/board-views/structure retaught
- story 24 -> in-progress, since frontmatter is now the only truth
- new iteration 38: fs mutation verbs + net.connect, the two capability
families no iteration owned
- new iteration 39: gofiber/fiber v3.5.0 parity study. The ledger called
CSRF/sessions unblocked by iteration 34's HMAC, but the runtime has no
source of randomness at all
- linkcheck skips .dev/.superpowers: 0 broken paths, 0 bad anchors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- runtime ids 91-95: net.read_dl/accept_dl/write_dl (per-call deadline,
nil/false = the EXPECTED timeout; ms<=0 = old behavior bit for bit),
net.listen_unix (unlink-before-bind, O_NONBLOCK on the listener —
probe-found: accept4's flag covers accepted sockets only), net.peer
- plane: one-op-per-park stays law — deadlines ride one per-shard
TIMEOUT tick (sentinel user_data) + post-CQE expiry sweep +
POLL_REMOVE tombstone; epoll's deadline scan grew the fd-park case;
fibers POOL instead of freeing mid-run (stale-CQE UAF); plain parks
zero park_deadline (no stale sleep deadlines)
- probe: all five seams verified on BOTH WO_IO backends (timeout
timing exact, peer round-trip, unix rebind)
- framework: parse_request grows first_ms/read_ms; serve_conn — the
keep-alive loop with deadlines where parked idle conns are LEGAL
(close-when-idle RETIRED); App.handle_conn exposes it; plain serve()
unchanged for simple apps
- web-app: app-owned accept_dl loop + ConnWorker actor per connection
(each builds its own App; cross-shard placement rides the DB actor);
WA_IDLE_MS knob; gate grows to 41 checks — two slow requests served
in PARALLEL, stalled client evicted at the idle deadline, slow-loris
torn at the read deadline (400)
- docs: story 35 -> done with banner; SQE/CQE design spec LANDED (was
the review doc); ledger rows (timeouts/unix/keep-alive/peer), graph
(NETSEAM cleared, KEEPAL done), builtin-surface rows, runtime
CODE-LOGIC section, board entry
- battery 13/13 fresh-built
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- one iteration by directive 2026-08-23: call() parks with typed reply
(envelope kinds 5/6 over the DB-RPC park), mailbox cap 1024 +
WO_T_ACTOR fail-fast, monitor(addr, msg) one-way, time.after
one-shot no-cancel
- story 34 resolved: C builtins sha1/sha256/hmac_sha256 over Bytes,
RFC vectors gated
- WS pure .wo: handler-owned upgrade (ws_accept + hijack sentinel),
frame codec over Bytes via 36's bitwise, two actors per connection
(sole-reader + sole-writer)
- chat sample: registry + room actors, python raw-RFC6455 gate —
cross-shard functional, 1k soak, SIGTERM drain, battery unchanged
- status PROPOSED — awaiting review before the plan
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- brings time.ticks builtin (id 84), bench sample + campaign driver
(scripts/db-bench.py), just db-bench/db-bench-quick recipes, first
baseline recorded, story 22 to done/, postgres study cards
- conflicts resolved: board in-progress table (iteration 36 +
framework rows kept, db-bench row now "22 landed"; dangling order
anchor repointed); story 36 moved back to in-progress/ (dir-rename
inference dragged it to done/ — 36 still awaits the manual pass)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- bench/baseline.json: 74 metrics from the first full campaign;
tolerances tuned by a two-run repeatability check (mix* 50%,
read/query 35%, rest 15% — rationale in _config)
- gate bites: --check mode; doctored copy fails, both real runs 74/0
- headline: durable seed 4.5k/s vs ram 297k/s (23's case); reads
O(table) at ~1.5k/s; mixread 1280 vs 21 ops/s single-vs-multi
(the arc's price); msgrate 13.4M vs 2.45M (mutex-inbox number)
- arc delta recorded in story 8; findings in sample README
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- worker DB builtins marshal to shard 0: requester-side slot encode
(VM heaps never read cross-shard), owner executes serialized in
adopt, reply unparks via new WO_PARK_INBOX park + envelope 3/4
- engine gains thread-agnostic slot entry points (insert_slots,
update_field_slot, val_encode/clone, wo_db_exec_req); traps and
messages byte-identical to the local path
- main.c: engine + replay boot BEFORE shards spawn; workers assert
rt.db/rt.wal NULL; busy shard adopts inbox once per slice
- latent stage-1 bug fixed: shared io_uring params static raced by
lazy worker init lost park wakes (~1/20 hangs); params per-vm,
short submit now fails loud
- new sample docs/examples/db-actor + just db-actor gate 8/0 (multi
x3, uring/epoll forced, single byte-exact, WAL replay pair);
ASan+TSan 6/6; full battery green
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- five-property map in marker doc: atomicity/durability/recovery
proven or held (18); concurrency control = stage 3's property;
space reclamation RAM done (slot reuse), disk = new story 32
- story 08: three stage-3 criteria (one commit per write RPC +
ack-after-owner-fsync, workers WAL-free + replay-before-serve,
no torn reads under TSan corpus); arc plan stage 3 carries them
- refine/32-wal-checkpoint.md: snapshot + truncate, bounded replay,
crash-during-checkpoint safe; four forks; after 23
- chain now stage 3 -> 22 -> 31 -> 24 -> 23 -> 32 in all 10 docs;
boards + seq bumps synced
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- 08 landed in discarded/ by mis-drop, swept into prior commit with
two dead links; corrected to stories/.../in-progress/ per intent
- 11 joins it (one arc, active slice)
- board doctrine: active stories bucket named; all links re-verified
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- io_uring is a must; epoll approach discarded (developer decision)
- plan superseded by shard-fiber-arc plan of record; banner + row in
plan/discarded.md; file kept as idea reference
- three live pointers repointed: status language-track row 8,
principles enforced-by, story 08 note
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- code-verified ready: arc stages 1+2 merged to master; stage 3
concrete in plan (tasks 7-8); rt.db set on primary only
(main.c), worker_late_init memsets rt — WO_T_DB hole real
- 22/23/24/31 stay in refine/: open forks, no bench harness,
no crypto builtins, chain-blocked
- links fixed both directions; board doctrine names hold/ bucket
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- iterations 18/25/26 marked hold in their spec + plan headers
- 25's story file removal committed; plan doc stays for resumption
- web-framework spec's relates-to flags 25 held
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Float full stack: literals (fraction/exponent; `0..10` still a range), f64
opcodes 34-41, @table column, WAL bit-exact replay, json fractions in and
shortest-round-trip out. IEEE-quiet — FDIV never traps where DIV does.
- Bytes: a wo_str with its own class id, so alloc/free/copy are shared but no
Text builtin accepts one; len/at/slice/eq/concat, base64 both ways, json
boundary as base64; TEXT_COPY preserves the kind.
- No implicit Int/Float mixing (WO-E201 in the typechecker, not the emitter,
which picks the opcode from one side and would misread the other).
- One IEEE deviation: float_cmp total order (NaN last, -0.0 == +0.0) for
indexes and order-by, keys canonicalized to match. `?Float` nil is a
reserved quiet NaN — the zero word is +0.0, WO_NIL_SCALAR's bits are -2.0.
- Renderer prefers fixed over exponential in 1e-6..1e21: pure shortest makes
a price of 900.0 read `9e+02`. One renderer for interp/json/float_to_text.
- Fixed en route: lexer double-counted the leading digit; is_scalar_shaped
took Float/Bytes as Int-shaped; Bytes ownership needed a shared heap-scalar
predicate or temps never dropped; order-by bit-compared negatives backwards.
- Iteration 17: `kind = "library"` (absent = program; bad value = WO-E109),
entry-less check mode retiring the `--emit` workaround, Go's `internal/` as
WO-E108 at the consumer's `use`. Driver-only; VM/.wob/GC untouched.
- Framework reorg: internal/{parse,serve}.wo; http/form.wo split out to keep
media_type/form_values public (parse.wo had grown public surface).
- Docs: link audit (97 -> 88 broken, conflict markers resolved, 2 duplicate
stories removed), 00-code-review verified 26/27, iterations re-sequenced.
- Also carries the pre-staged pub(read)/using/#if work from the index.
- Gates: corpus 103/0, test_wal 156/0, web-app 26/0, oop-accept ALL MET.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- developer directive: pending iteration IDs now ARE the priority order;
LANDED iterations keep historical numbers (code comments and commit
history cite them — records, not a queue); 8/11 (the half-landed
arc), 17 (parked, artifacts on a branch), 18 (next, artifacts named)
also frozen
- mapping (recorded in 00-story): 19<-20 Float+Bytes, 20<-9c attach,
21<-9d keypair, 22<-9e benchmarks, 23<-9f io_uring WAL, 24<-19 chat,
25<-10 services, 26<-12 blue-green, 27<-9g query corpus,
28<-14 skillhost, 29<-13 metaprogramming
- 11 story files renamed; every doc reference re-numbered (word-boundary
sweep for the lettered 9x ids, phrase-level for numeric ones); the
iterations table rewritten with Seq == priority and "(was N)" notes;
story-scoped link check: zero broken
- merge-recovery folded in: the partial master merge had dropped the
chat story, the fibers exploration note, the arc spec+plan, the
framework-v2 plan, and the iteration-17 spec+plan — all restored from
their branches and renumbered consistently
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- done/ (11): 1, 2, 3, 4, 6, 7, 7b, 9, 9b, 15, 16 — landed iterations
(9/9b remainders live in the post-12 drain list, not in the files)
- refine/ (8): 9c, 9d, 9e, 9f, 9g, 11, 13, 14 — everything marked
"no spec yet / brainstorm before planning"
- root keeps: 00-story (index), 05 (partial, plan 8 open), 8/10/12
(specs or plans exist), 17 (parked, spec+plan approved), 18 (next)
- every cross-reference re-pathed and VERIFIED resolving: board, specs,
plans, employee-list README, story table, intra-story links (moved
files' relative links deepened one level; done/7b's 9e pointer now
crosses to refine/)
- pre-existing dead link noted, not touched: refine/11-fibers.md points
at docs/plan/exploration/fibers/00-fibers.md which does not exist
(predates the move)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- docs/00-dependency-graph.md: three mermaid graphs — story iterations
(hard edges only; 18 is the only spec-approved node with all
prerequisites green), framework v1 ledger items (three recurring
gates: net seams, crypto fork, iterations 8/11; nine slices startable
today in any order), framework v2 internals (cache/flags independent,
transaction{} is the critical path, jobs compose on it)
- maintenance rule: node classes update in the same change as board rows
- board links the graph up top; spec 18 banner -> APPROVED, plan next
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- framework README core checklist expanded into the v1 STATUS LEDGER:
seven categories (transport, routing, request/response, context &
middleware, storage integration, security, crypto), every item
marked done / partial-with-named-gap / candidate / parked-behind-8-11
/ needs-runtime-seam
- verified before labeling: BODY_MAX caps headers AND body (size limits
done); net has no timeout or unix-socket or peer-address surface
(runtime seams); language has NO bitwise operators, so SHA/HMAC/CRC32
must be C runtime builtins or bit ops land first (fork to brainstorm);
radix routing waits for 9e to measure the linear scan first
- crypto hard stop recorded: HS256 unlocks and nothing past it
- memory-rich features relabeled FRAMEWORK V2 = iteration 18 (story +
spec banners + board rows); v1 gaps land as slices per the ledger
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Part A transaction: one wal_commit at block end over the existing
staged batch; reads see own writes (RAM stays authoritative); trap
unwinding out = abort (undo list: insert->remove, update/delete->
pre-image, captured before RAM apply, txn-only cost); try inside
keeps the block alive; WO-E110 lexical nesting, WO_T_DB dynamic;
no new opcodes, no .wob bump (internal builtins + catch-frame-shaped
abort marker); E108/E109 stay reserved for parked 17
- Part B: cache.wo (ttl_ms/cap, lazy time.now-ms expiry, FIFO over LRU
with the tradeoff stated, Text values via json); flags.wo (@table
wf_flags, on as Int 0/1 - Bool columns unproven, read-through map,
set updates table+map); jobs.wo (@table wf_jobs, enqueue composes
with transaction, JobRunner interface, App.jobs(take r, budget),
Dispatcher.idle() called post-accept PRE-PARSE - deterministic for
the SIGKILL durability proof, unlike after-response)
- web-app demo: transactional order+confirm enqueue, GET /jobs count,
POST /flags/:name with a flag-gated header on the product list
- gate: SIGKILL-after-201/restart/drain proof + flags persistence;
corpus carries transaction-commit/abort + WO-E110 + cache-ttl
(stamps injected, no sleeps)
- board row 18 -> spec written, awaiting review
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Board row 16 -> landed (web-app 14/0), pending row removed; story header
records the landing + the two as-built discoveries (idle-keep-alive
starvation policy; the two compiler gaps the chain exposed and fixed);
README gains the framework+web-app sample entry; plan checkboxes ticked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- error catalog: WO-E106 (dependency fetch/shape failures, one code, message
names dep + step) and WO-E107 (dep/local module-name collision) rows.
- README: a Dependencies subsection under the manifest docs — [deps] syntax,
.wo-deps/wo.lock behavior, offline-when-locked, --update-deps, flat-only.
- board: iteration 15 row -> landed (deps-accept 8/0), pending row removed;
story 15 header records the landing; 08-project-structure notes
.wo-deps (gitignored) + wo.lock (committed); plan checkboxes all ticked.
(One self-inflicted casualty during this task, restored from git before
commit: a buggy doc-edit script truncated 08-project-structure.md; the file
was recovered intact and the intended one-liner applied by hand.)
Gates at closeout: deps-accept 8/0, woc-test 540/0, oop-e2e 87/0,
log-watcher 7/0, employee 8/0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5 tasks, words-only per house rule, each with its verify step: (1) manifest
grows the [deps] section + one-line inline-table value (only under [deps]);
(2) resolver — git-binary fetch into .wo-deps/, wo.lock pinning, warm-path
offline guarantee, drift diagnostic, --update-deps, guard rails (transitive
refusal, non-writeonce dep, name collision WO-E107, all fetch failures
WO-E106); (3) multi-root discovery + module_of prefixing dep roots by dep
name + entry restricted to the app's own files; (4) scripts/deps-accept.sh
gate over file:// remotes (8 checks, network-free) + just recipe; (5) docs
closeout (catalog E106/E107, README deps subsection, board/story/structure).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brainstorm outcome (forks locked with the developer):
- TLS: proxy-terminated (nginx/caddy gives browsers TLS+ALPN+h2; the
framework speaks HTTP/1.1 behind it) — zero TLS in the toolchain, no
doctrine fight; homegrown TLS refused outright.
- Dependencies: a real mini package manager — wo.toml [deps] with exact-rev
git deps, wo.lock, .wo-deps cache, `use <dep>` as a module root; fetch by
shelling to the git binary (no network code in woc); flat-only v1.
- HTTP/2: v1 is HTTP/1.1 keep-alive; h2c is the parked successor behind
iterations 8/9f/11 (multiplexing needs a scheduler to pay off); the
bytes/buffer type rides with it, not v1.
- Handler model: no function values by doctrine, so Handler/Middleware are
structural interfaces (ICALL dispatch, WO-E205-checked); middleware returns
?Resp and rides the shipped ?T narrowing.
- Incubation: framework at docs/examples/writeonce-framework/, consuming
storefront at docs/examples/web-app/ importing it THROUGH [deps] — the
sample exercises fetch -> lock -> build -> serve -> durable-restart.
- Iteration 10 relationship: service blocks later LOWER ONTO this library.
Files: specs/2026-08-18-web-framework-design.md (A deps normative, B
framework normative, C h2c parked); stories 15-deps-package-manager.md +
16-web-framework.md; roadmap + board rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spec's §8 migration table, applied:
- 00-principles.md P3: "@gc is a per-class opt-in, reference-counted" ->
GC-ness is inferred; incremental per-shard mark-sweep in budgeted slices;
still no global pause by construction.
- OOP spec: decision-table GC row -> inferred (hybrid rule named); §3 rule 5
-> traced classes alias freely, which classes is inferred; §4 memory model
-> the RC + Bacon-Rajan paragraph replaced by tracing (snapshot roots,
Yuasa barrier, born-black, budgeted slices); header rc comment -> union'd
sweep link; mixing rule restated for tracing.
- 00-wob-format.md: header says version 4; opcodes 27-28 -> reserved (loader
rejects); the owned-temporary rule's @gc exclusion restated for tracing.
- 08-builtin-surface.md: the push RC_INC special case and the set(m,k,v)
retention gap DELETED — neither exists without RC; the corpus cycle is
collected by tracing.
- story 07b: status -> LANDED 2026-08-18 (with the historical note kept);
board: 7b row ✅ (supersedes iteration 2's RC memory model), pending row
removed.
- gc-cycle README: Phase 3 flipped to landed (the ring runs, is reclaimed,
ASan-clean; the ?Node RC_DEC-on-nil trap no longer exists); the barrier
prose corrected to the as-built design (snapshot-at-beginning + deletion
barrier + born-black, not per-slice root re-reads).
- plan 2026-08-18: all checkboxes ticked + a completion banner recording the
four deviations from the plan as written.
(Error catalog was already amended with the keyword-removal commit: WO-E104
added, WO-W201 retired.)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Analyzed the full 131-file docs tree (4 parallel classifiers) against the
shipped woc/wovm toolchain. Removed 21 stale docs, kept all intentional
history (Rust-track plans/done, the runtime/database design series cited by
current specs, syscall/postgres/assembly/c-runtime studies, discarded/
learnings). Deleted:
- old-runtime "front door": writeonce-pl.md, runtime/wo-language.md
(pitched the Rust wo runtime -- REST/LiveView/SQL+Cypher -- as the current
language; contradicted the new README)
- v1 design set: 02-recovery, 03-data, 04-ui, 05-datalayer,
06-markdown-render, 07-ssl; runtime/database/05-go-sdk
- future-scope/ai-agents-content-management (unfinished old-runtime CMS)
- the ##ui/.htmlx LiveView frontend track (product decision to abandon):
9 plan/exploration/ui/*, plan/14-mvc-ui-implementation,
superpowers/plans/2026-08-01-ui-htmlx-live; 13d pricing-UI board row
Tree left link-clean: 46 dead links to the removed docs neutralized to plain
text or deleted as pure see-also bullets across 20 kept docs; whole-tree
link-resolving scan reports zero links to any deleted file. Removal recorded
in discarded.md; board Frontend section + project-structure tree updated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Task 6 note: db fixtures live in existing corpus kinds; crash battery
proven at unit level; select fixtures wait on 9b's read surface
- board row updated
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
