From aa13b2125fcde51bfb2b708315b2ca2d319ef114 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Sun, 30 Aug 2026 04:56:16 +0200 Subject: [PATCH] refactor(porch-store): re-scope porch 1 to the limiter, revert idempotency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - idempotency built, reviewed, then reverted WHOLE to the tag archive/porch-idempotency. Not a design failure: it passed its gates. It provokes a C-runtime SIGSEGV in wo_arena_alloc/wo_str_new under concurrent call()-parked callers - the evidence for that attribution: over ten gate runs every failure was an idempotency leg and none was the limiter's, which drives the same pool through the same call/park machinery. The begin arm has 5x the allocation sites inside receive and moves a whole Req plus a Handler through the mailbox - before the split the suite reported 0 to 6 failures run to run; after it, five consecutive runs at 56 checks, 0 failures - PoolMsg loses digest/req/handler, and NullHandler/dummy_req/fresh_req go with them — every rate-limit count used to allocate a throwaway Req it never read - IdempotencyKey is KEPT and commented: the schema is settled and the digest-as-column decision cost a review round to get right - the limiter's saturation 503 has no leg of its own now (§19 drove Idempotent). Stated in the README rather than papered over — a deterministic leg needs a slow actor, and only the reverted arm was - new: porch 9 (idempotency, on hold) and language 41 (the arena crash, with the reproduction harness and the evidence that localises it) Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit 79e6da4465133dc555e913c960d544ef1c7bedd8) --- docs/examples/porch/README.md | 4 +- docs/examples/porch/middleware/idempotent.wo | 121 ---- docs/examples/porch/middleware/keypool.wo | 211 +----- docs/examples/porch/middleware/store.wo | 21 + ...29-porch-store-backed-middleware-design.md | 11 + scripts/web-app-accept.sh | 638 ------------------ 6 files changed, 71 insertions(+), 935 deletions(-) delete mode 100644 docs/examples/porch/middleware/idempotent.wo mode change 100755 => 100644 scripts/web-app-accept.sh diff --git a/docs/examples/porch/README.md b/docs/examples/porch/README.md index 4310ca3..b4b5487 100644 --- a/docs/examples/porch/README.md +++ b/docs/examples/porch/README.md @@ -156,8 +156,8 @@ first (pure `.wo` cannot express it yet). | Item | State | | --- | --- | -| Rate limiting (fixed window, durable) | ✅ counting serializes through a per-key actor pool (`middleware/keypool.wo`, `limiter.wo`) — no handler-fiber read-modify-write left to lose an increment. Gate-proven: exact count under genuine concurrency (30 parallel requests, no lost increments), WAL-durable restart still limiting, and `trust_proxy`'s peer fallback — [porch 1](../../stories/porch/01-store-backed-middleware.md). Saturation failing closed (503) shares `limiter.wo`'s own `try/catch` code path with idempotency's, but is gate-proven only via `Idempotent`/`pool_begin`'s own saturation leg in `scripts/web-app-accept.sh` — no leg drives `Limiter`'s own 503 arm directly | -| Idempotent replay of unsafe requests | ✅ the pool actor runs the route's `Handler` itself (`middleware/idempotent.wo`), so a duplicate blocks in the actor's mailbox until the owner's row commits — no in-flight heuristic, no window where a duplicate can see "nothing yet". Gate-proven: byte-identical replay, digest-mismatch refusal (422), concurrent duplicates never double-executing, a transient 5xx never replayed (solo or concurrent), ephemeral rows not leaking, and pool saturation failing closed (503) — [porch 1](../../stories/porch/01-store-backed-middleware.md) | +| Rate limiting (fixed window, durable) | ✅ `Limiter` middleware over a sharded actor pool — exact counting under 30 genuinely-parallel clients, WAL-durable across a SIGTERM restart, `trust_proxy` off by default with a `net.peer` fallback. **One gap, stated rather than hidden:** the fail-closed 503 on a saturated pool is correct by construction (same `try`/`catch` as the arm that was gate-proven) but has no leg of its own — saturating the count arm deterministically needs a slow actor, and only the reverted idempotency arm was slow. The proof lives in `archive/porch-idempotency` | +| Idempotent replay of unsafe requests | ⏸ **built, reviewed, then reverted 2026-08-30.** Not a design failure: the pool actor ran the route handler inside its own `receive` so a duplicate waited in the mailbox, and it passed its gates. It provoked a C-runtime SIGSEGV in `wo_arena_alloc`/`wo_str_new` under concurrent `call()`-parked callers. Whole in `archive/porch-idempotency`, which doubles as the reproduction harness. Blocked on the runtime fix | | Transaction-per-request middleware (commit on 2xx, roll back otherwise) | ⏸ **v2** — needs iteration 18's `transaction { }` | | Cancellation → rollback | ⏸ arc landed; still needs v2's `transaction { }` (iteration 18) | | Migration generation + review workflow | ⬜ recorded future story (script-based destructive migrations) | diff --git a/docs/examples/porch/middleware/idempotent.wo b/docs/examples/porch/middleware/idempotent.wo deleted file mode 100644 index bb528c0..0000000 --- a/docs/examples/porch/middleware/idempotent.wo +++ /dev/null @@ -1,121 +0,0 @@ --- porch/middleware/idempotent.wo — idempotency, actor-run. --- Iteration 1 of the porch track, porch-store task 4. --- --- Rebuilt, not patched: the old before/after shape ran the handler and --- stored the response afterward, so two simultaneous duplicates both --- missed and both executed — before() has nothing to find until after() --- runs, which is too late for the request that is racing it. The fix is --- that the ACTOR runs the handler: Idempotent wraps the route's own --- Handler and hands both the request and that handler to the pool. A --- duplicate for the same key then simply waits in the actor's mailbox --- (one message at a time) and is dequeued once the owner's receive has --- already committed the row — no in-flight heuristic needed, because --- there is no window where a duplicate can see "nothing yet". --- --- `call`'s reply is a copyable scalar only (WO-E226): keypool.wo's kind-2 --- arm answers with the SAME pool_pack(count, remaining_ms) encoding kind-1 --- uses, never the response itself. The response travels through the --- @table instead — the actor stores it, this file reads the same row --- back and builds the Resp from it, so owner and duplicate answer with --- byte-identical bytes structurally, not by careful bookkeeping. -use http -use json - --- Idempotent wraps a route's Handler. Registration: Idempotent { key_header: --- "Idempotency-Key", pool: p, inner: CreateOrder {} } in place of the bare --- handler in app.post(...) -- key_header is matched case-insensitively --- (req.headers keys are lowercased on read, so any case here works). Only --- the response allowlists content-type, location, etag, cache-control for --- replay (never Set-Cookie, never Date) — cookies arrive in porch 2. -pub class Idempotent { - key_header: Text -- e.g., "Idempotency-Key" (matched case-insensitively) - pool: Pool - inner: Handler -- the real route handler; the actor runs this - include_body: Bool = true -- digest method+path+body, refuse a key reused with a different one - ttl: Int = 86_400_000_000 -- 24h in µs, lazy-expired on access - - fn handle(req: Req) -> Resp { - -- req.headers keys are lowercased on read (internal/parse.wo); normalise - -- key_header the same way, or a documented `key_header: "Idempotency-Key"` - -- (capitalised, as app authors are told to write it) NEVER matches and - -- this middleware silently falls through to self.inner.handle(req) below - -- on every request -- idempotency disabled, no 503, no log. - let name = to_lower(self.key_header); - let header_val = req.headers[name]; - if header_val == nil { return self.inner.handle(req); } - - let key = "idem:${name}:${header_val}"; - -- method+path scope the digest unconditionally: with include_body false - -- a bare "" digest would match ANY other request under this same key, - -- letting a different route/method replay this one's stored response. - let digest_input = "${req.method}|${req.path}"; - if self.include_body { digest_input = "${digest_input}|${req.body}"; } - let digest = base64_encode(bytes_slice(sha256(bytes_of_text(digest_input)), 0, 16)); - - let raw = try pool_begin(self.pool, key, digest, self.ttl, req, self.inner) catch (e) { - print_err("idempotent: pool_begin trapped: ${e.msg}"); - nil - }; - if raw == nil { - let r = Resp { status: 503, headers: {}, body: "{\"error\":\"idempotency store saturated\"}" }; - set_header(r, "content-type", "application/json"); - set_header(r, "retry-after", "1"); - return r; - } - - let outcome = raw / 1_000_000_000; - if outcome == 2 { - -- Same key, a different request: refuse rather than serve the - -- other request's response. - let r = Resp { status: 422, headers: {}, body: "{\"error\":\"idempotency key reused with a different request\"}" }; - set_header(r, "content-type", "application/json"); - return r; - } - - -- Outcome 1: the bare key names a durable (2xx/3xx) replay target -- - -- this file never deletes it; it is the whole point of a durable row. - -- Outcome 3: this call's own 4xx/5xx answer lives under a row keyed - -- by a nonce (the reply's low digits) that nobody else's message - -- for this same bare key ever writes to -- rebuild that exact key - -- rather than reading the bare one, so a race with a LATER message - -- for this key (which never touches this row) can't hand back the - -- wrong response. - let lookup_key = key; - if outcome == 3 { lookup_key = "${key}#eph:${raw % 1_000_000_000}"; } - let hits = from k in IdempotencyKey where k.key == lookup_key take 1 select k; - if len(hits) == 0 { return server_error(); } - let row = hits[0]; - let stored = json.decode(row.response) as IdempotentStoredResp; - if stored == nil { return server_error(); } - -- `.. ""` forces a fresh, independently-owned Text for every key/value - -- copied out of the decoded record: json.decode's Text values do not - -- survive being handed onward as-is once the decoded record itself - -- goes out of scope (a stale row read back corrupted mid-response - -- otherwise) — concat is documented to always allocate new owned text. - let hdrs: map = {}; - for k, v in stored.headers { hdrs[k .. ""] = v .. ""; } - let result = Resp { status: stored.status, headers: hdrs, body: stored.body .. "" }; - if outcome == 3 { - -- Safe to delete here, unlike the shared bare-key row rounds 1/2 - -- removed: the nonce that names this row was never handed to - -- anyone but this one call() reply, so no other request -- a - -- duplicate, a retry, anything -- can ever construct this exact - -- key to read it. Deleting it removes the leak AND the - -- nonce-wraparound collision (time.ticks() % 1_000_000_000 repeats - -- every ~1000s; a lingering row from an earlier failed attempt - -- landing on the same nonce would otherwise be there to collide - -- with, or worse, poison the actor's OWN unguarded insert on the - -- next failure for this key). - delete row; - } - return result; - } -} - --- JSON shape of IdempotencyKey.response. Allowlisted headers only: --- content-type, location, etag, cache-control, never Set-Cookie or Date. -typedef IdempotentStoredResp = { - status: Int, - headers: map, - body: Text -} diff --git a/docs/examples/porch/middleware/keypool.wo b/docs/examples/porch/middleware/keypool.wo index 43472c8..f66bd29 100644 --- a/docs/examples/porch/middleware/keypool.wo +++ b/docs/examples/porch/middleware/keypool.wo @@ -17,22 +17,21 @@ -- type, so a second encoding is not an option. use time -use http -use json -- To a pool actor. kind 1 = count (this file); kind 2 = begin (Task 4 -- fills in the arm — the fields below are already shaped for it: the -- bare idempotency key travels in `key`, the body digest in `digest`, -- and the actor runs `handler` against `req` itself so a duplicate waits -- in the mailbox rather than needing a held reply). +-- To a pool actor. `kind` is kept even though only one kind exists today: +-- idempotency's `kind: 2` arm was built, reviewed and then REVERTED (see +-- archive/porch-idempotency), and it will come back. Adding a second kind is +-- a field and an `if`, not a redesign. class PoolMsg { - kind: Int - key: Text - limit: Int -- count: max requests per window - window: Int -- count: window size, µs - digest: Text -- begin: sha256(method|path|body), Task 4 - req: Req -- begin: the request, Task 4 - handler: Handler -- begin: the route's handler, invoked inside receive, Task 4 + kind: Int + key: Text + limit: Int -- count: max requests per window + window: Int -- count: window size, µs } -- What the limiter reads back from a count. `allowed` and `limit` are @@ -46,45 +45,23 @@ class Verdict { } -- PoolMsg requires `req`/`handler` on every construction (an actor --- message's fields are all required, like RoomMsg's `writer` in --- docs/examples/chat/main.wo). A count message has no request to run, so --- it fills those two with an inert placeholder — same shape as chat's --- dummy_writer() for RoomMsg's shutdown message. -class NullHandler { - fn handle(req: Req) -> Resp { - return Resp { status: 500, headers: {}, body: "" }; - } -} +-- NOTE: this file used to carry NullHandler, dummy_req() and fresh_req(). +-- They existed ONLY because PoolMsg had to carry a Req and a Handler for +-- idempotency's kind-2 arm, which meant every rate-limit count allocated a +-- throwaway Req (four maps) it never read. With that arm reverted the +-- placeholders go too, and counting stops paying for a feature it never +-- used. They are preserved with the arm in archive/porch-idempotency. -fn dummy_req() -> Req { - return Req { - method: "", path: "", params: {}, query: {}, headers: {}, - body: "", principal: "", ctx: {}, conn: 0 - 1 - }; -} - --- A live Req arriving at a Handler is a borrow (Handler.handle's signature --- fixes that, in router.wo — not this file's to change): its map fields --- are references that cannot outlive the caller's scope, so forwarding --- them as-is into an actor message is refused (WO-E222 — the same --- aliasing rule Pool's own doc comment above describes). Copying each map --- field into a brand-new map, then building a brand-new Req from that plus --- the plain scalars, produces a value with no other referrer — the same --- shape dummy_req() already sends, just carrying the real request. -fn fresh_req(r: Req) -> Req { - let params: map = {}; - for k, v in r.params { params[k] = v; } - let query: map = {}; - for k, v in r.query { query[k] = v; } - let headers: map = {}; - for k, v in r.headers { headers[k] = v; } - let ctx: map = {}; - for k, v in r.ctx { ctx[k] = v; } - return Req { - method: r.method, path: r.path, params: params, query: query, - headers: headers, body: r.body, principal: r.principal, ctx: ctx, - conn: r.conn - }; +-- Packs (count, remaining-ms-in-window) into one Int: count * 1e9 + +-- remaining_ms, remaining_ms clamped to stay under 1e9 (~11.5 days — +-- far past any realistic rate-limit window). That clamp only blurs the +-- advisory reset header on an absurdly long window; it never touches the +-- count, which is the correctness-critical half. +fn pool_pack(count: Int, remaining_ms: Int) -> Int { + let r = remaining_ms; + if r < 0 { r = 0; } + if r >= 1_000_000_000 { r = 999_999_999; } + return count * 1_000_000_000 + r; } -- One actor per shard. Reads the row for the key, decides, and writes the @@ -94,94 +71,16 @@ fn fresh_req(r: Req) -> Req { -- row for the new window — the stale row is retired, not mutated). class KeyActor { fn receive(msg: PoolMsg) -> Int { - if msg.kind == 2 { - -- Task 4: idempotency begin. msg.window carries the TTL here (both - -- are µs durations; kind 1 has no use for a TTL and kind 2 has no - -- use for a window, so the one field serves both). A miss runs - -- msg.handler right here, inside receive, so a duplicate already - -- queued behind this message dequeues to a settled row instead of - -- a race. - let now = time.ticks(); - let hits = from k in IdempotencyKey where k.key == msg.key take 1 select k; - - if len(hits) > 0 { - let stored = hits[0]; - if now - stored.created_at > msg.window { - -- Expired: lazy delete (no sweeper exists), fall through to miss. - delete stored; - } else if stored.digest == msg.digest { - -- A row lives under the BARE key only when it is durable (see - -- below) -- an ephemeral one never does -- so any hit here is - -- already a stable, valid replay target. - return pool_pack(1, 0); - } else { - -- Same key, a different request: refuse rather than serve the - -- other request's response. - return pool_pack(2, 0); - } - } - - -- Miss: fresh key, an expired row just deleted, or the prior - -- attempt (if any) was ephemeral and so is invisible to the - -- bare-key lookup above -- all three run the handler fresh. - let resp = msg.handler.handle(msg.req); - -- Allowlist, not denylist: an allowlist fails safe when Resp grows a - -- new header later (excluded from replay until reviewed, never - -- replayed by accident from day one). content-type alone dropped - -- Location off every redirect() and any Etag/Cache-Control a handler - -- set; matched case-insensitively since set_header writes the name - -- verbatim (a handler using "Content-Type" was silently losing it). - let hdrs: map = {}; - for hk, hv in resp.headers { - let lhk = to_lower(hk); - if lhk == "content-type" or lhk == "location" or lhk == "etag" or lhk == "cache-control" { - hdrs[lhk] = hv; - } - } - let stored_json = json.encode(IdempotentStoredResp { - status: resp.status, headers: hdrs, body: resp.body - }); - if resp.status >= 200 and resp.status < 400 { - -- 2xx/3xx: a real answer worth replaying for the TTL, stored - -- under the bare key -- one stable row per key, unguarded same - -- as kind 1's own insert (this actor is the only writer for - -- this key; a @unique violation here would mean something is - -- genuinely wrong, not a race to paper over -- a swallowed - -- failure would answer "stored" for a response never written). - insert IdempotencyKey { - key: msg.key, response: stored_json, created_at: now, digest: msg.digest - }; - return pool_pack(1, 0); - } - -- 4xx/5xx: never a replay target, so it does NOT go under the bare - -- key -- a bare-key row is memoryless (this file's own doc above), - -- but a SHARED, mutable row is not: a second message racing the - -- first could delete-and-replace it before the first caller's own - -- middleware-side read (necessarily outside receive -- WO-E226, - -- a Resp cannot ride the mailbox) ever runs, so the FIRST caller - -- could read back the SECOND caller's answer. Every miss instead - -- gets its own row, keyed by a nonce carried back in the scalar's - -- low digits (the same slot pool_pack's remaining_ms uses for - -- kind 1) so idempotent.wo can reconstruct the exact same key and - -- read only ever what THIS call produced -- immune to any other - -- message touching this bare key, ever. That same unguessability - -- (the nonce is never handed to anyone but this one call() reply) - -- is also why idempotent.wo deletes this row right after reading - -- it: nothing else can ever construct this exact key, so nothing - -- else is deleted out from under. Without that delete, the row - -- would linger forever (no sweeper exists) AND time.ticks() % 1e9 - -- wraps every ~1000s, so a later failed attempt for the SAME - -- bare key landing on the same nonce would collide with it -- - -- reintroducing a stale-replay risk on wraparound, or poisoning - -- this actor's own unguarded insert above. Deleting it removes - -- both, not just the storage growth. - let nonce = now % 1_000_000_000; - insert IdempotencyKey { - key: "${msg.key}#eph:${nonce}", response: stored_json, created_at: now, digest: msg.digest - }; - return pool_pack(3, nonce); - } + -- Only kind 1 exists today. The `kind: 2` arm — idempotency, where the + -- actor ran the route handler inside this receive so a duplicate waited + -- in the mailbox — was built, reviewed and then REVERTED. It is whole in + -- the tag archive/porch-idempotency, which doubles as the reproduction + -- harness for the C-runtime crash that caused the revert: a SIGSEGV in + -- wo_arena_alloc / wo_str_new under concurrent call()-parked callers. + -- That arm allocated 5x what this one does inside receive and moved a + -- whole Req plus a Handler through the mailbox; over ten gate runs every + -- failure was one of its legs, and none were this one's. -- kind 1: count. let now = time.ticks(); let hits = from c in RateLimitCounter where c.key == msg.key take 1 select c; @@ -209,22 +108,6 @@ class KeyActor { } } --- Packs (count, remaining-ms-in-window) into one Int: count * 1e9 + --- remaining_ms, remaining_ms clamped to stay under 1e9 (~11.5 days — --- far past any realistic rate-limit window). That clamp only blurs the --- advisory reset header on an absurdly long window; it never touches the --- count, which is the correctness-critical half. -fn pool_pack(count: Int, remaining_ms: Int) -> Int { - let r = remaining_ms; - if r < 0 { r = 0; } - if r >= 1_000_000_000 { r = 999_999_999; } - return count * 1_000_000_000 + r; -} - --- One actor address per slot. `multi actor PoolMsg` does not parse (a --- `multi`'s element type is one token) — chat/main.wo's RoomRef wraps an --- actor handle in a one-field class for exactly this reason, mirrored --- here as PoolSlot. class PoolSlot { a: actor PoolMsg } @@ -308,10 +191,7 @@ pub fn pool_select(pool: Pool, key: Text) -> actor PoolMsg { -- actor's scalar reply into the Verdict the limiter reads. pub fn pool_count(pool: Pool, key: Text, limit: Int, window: Int) -> Verdict { let a = pool_select(pool, key); - let raw = call(a, PoolMsg { - kind: 1, key: key, limit: limit, window: window, - digest: "", req: dummy_req(), handler: NullHandler {} - }); + let raw = call(a, PoolMsg { kind: 1, key: key, limit: limit, window: window }); let count = raw / 1_000_000_000; let remaining_ms = raw % 1_000_000_000; return Verdict { @@ -322,23 +202,6 @@ pub fn pool_count(pool: Pool, key: Text, limit: Int, window: Int) -> Verdict { }; } --- The begin accessor idempotent.wo calls: hides pool_select/call the same --- way pool_count does. Returns the raw packed outcome — 1 means the --- BARE key names a durable (2xx/3xx) replay target; 2 means a digest --- mismatch (422, nothing to read); 3 means this call's own 4xx/5xx --- answer lives under "${key}#eph:${raw % 1_000_000_000}" instead -- --- the low digits of the reply are that row's own nonce, not a window --- size (kind 1's use of the same slot), so idempotent.wo can rebuild --- the exact key and read only ever what THIS call produced. Never 0: --- idempotent.wo's own catch-and-log-nil handler cannot tell a literal 0 --- reply apart from a trapped call, so the encoding avoids it on --- purpose. A trapped call (a saturated mailbox) propagates to the --- caller uncaught, same as pool_count -- the middleware's own --- try/catch answers 503. -pub fn pool_begin(pool: Pool, key: Text, digest: Text, ttl: Int, req: Req, handler: Handler) -> Int { - let a = pool_select(pool, key); - return call(a, PoolMsg { - kind: 2, key: key, limit: 0, window: ttl, - digest: digest, req: fresh_req(req), handler: handler - }); -} +-- NOTE: pool_begin() lived here — the accessor idempotent.wo called to run a +-- request through the actor. Reverted with the kind-2 arm; whole in the tag +-- archive/porch-idempotency. diff --git a/docs/examples/porch/middleware/store.wo b/docs/examples/porch/middleware/store.wo index 0a8246a..b95a829 100644 --- a/docs/examples/porch/middleware/store.wo +++ b/docs/examples/porch/middleware/store.wo @@ -12,6 +12,27 @@ class RateLimitCounter { window: Int } +-- ============================================================================ +-- KEPT DELIBERATELY, UNUSED TODAY. +-- +-- Nothing in porch reads or writes this table right now. The idempotency +-- middleware that did was reverted on 2026-08-30 — not because the design was +-- wrong (it was built, reviewed and works) but because it provoked a C-runtime +-- crash: a SIGSEGV in wo_arena_alloc / wo_str_new under concurrent +-- call()-parked callers allocating heavily inside an actor's receive. Over ten +-- gate runs every failure belonged to an idempotency leg and none to the rate +-- limiter's, which drives the same pool through the same machinery but +-- allocates a fifth as much. +-- +-- The table stays because the schema is settled and re-adding it would be +-- churn, not design: `digest` as its own column (never folded into the key, or +-- "same key, different body" becomes undetectable) is the one decision that +-- cost a review round to get right. The middleware, its actor arm and its gate +-- legs are whole in the tag `archive/porch-idempotency`, which is also the +-- reproduction harness for the runtime bug. +-- +-- If the runtime bug is fixed and idempotency is NOT resumed, delete this. +-- ============================================================================ -- Idempotency: stored response for replay. -- Key format: "idem:keyheader" or "idem:keyheader:sha256(method|path|body)" -- Response = JSON-encoded Resp {status, headers, body} (headers allowlist: diff --git a/docs/superpowers/specs/2026-08-29-porch-store-backed-middleware-design.md b/docs/superpowers/specs/2026-08-29-porch-store-backed-middleware-design.md index b02369e..1d772ea 100644 --- a/docs/superpowers/specs/2026-08-29-porch-store-backed-middleware-design.md +++ b/docs/superpowers/specs/2026-08-29-porch-store-backed-middleware-design.md @@ -1,5 +1,16 @@ # porch 1 — store-backed middleware: rate limiting and idempotency +> **Status 2026-08-30 — half of this spec shipped, half is on hold.** +> The rate limiter landed and is stable. Idempotency was implemented against +> this design, passed review, and was then reverted whole to the tag +> `archive/porch-idempotency` because it provokes a C-runtime SIGSEGV in +> `wo_arena_alloc`/`wo_str_new` +> ([language 41](../../stories/language-runtime-database/41-actor-arena-crash.md)). +> **Nothing below is retracted** — the design is sound and was proven by a +> working implementation. It is waiting on the runtime, and it is tracked as +> [porch 9](../../stories/porch/09-idempotent-replay.md). + + Design settled 2026-08-29. Implements [porch 1](../../stories/porch/01-store-backed-middleware.md). diff --git a/scripts/web-app-accept.sh b/scripts/web-app-accept.sh old mode 100755 new mode 100644 index 4dd619e..ba2e35a --- a/scripts/web-app-accept.sh +++ b/scripts/web-app-accept.sh @@ -736,644 +736,6 @@ else bad "limiter-compile" "$(printf '%s' "$lp_out" | head -1)" fi -# ---- 18. porch-store task 4: idempotency runs the handler inside the actor -- -# Same flattening trick as the keypool/limiter legs. Idempotent wraps a -# SLOW route handler (500ms) so two genuinely-parallel duplicates actually -# overlap in the pool's mailbox. The handler's side effect (ExecMark) is -# a real @table row count read back over GET /execs -- never a log line, -# per the brief. One server serves all three legs with distinct keys, so -# the exec count accumulates 1 -> 2 -> 3 across them. -IP="$W/idempotent-check" -cp -r "$ROOT/docs/examples/porch" "$IP" -rm -f "$IP/wo.toml" -rm -rf "$IP/target" -cat >"$IP/idempotent_check_main.wo" <<'WOEOF' -use net -use env -use http -use router -use middleware -use time - -@table(name: "exec_marks") -class ExecMark { - n: Int -} - --- a deliberately slow handler: the concurrency leg's workload. Records --- one row per REAL execution so a duplicate that wrongly ran it too --- shows up as a row-count of 2, never as a log line. -class SlowHandler { - fn handle(req: Req) -> Resp { - insert ExecMark { n: 1 }; - let n = len(from e in ExecMark select e); - time.sleep(500); - return ok_json("{\"echo\":\"${req.body}\",\"exec\":${n}}"); - } -} - -class ExecCount { - fn handle(req: Req) -> Resp { - let n = len(from e in ExecMark select e); - return ok_json("{\"count\":${n}}"); - } -} - -@table(name: "flaky_marks") -class FlakyMark { - n: Int -} - --- reviewer finding, task 4 follow-up: a transient 5xx must not be cached --- for the TTL -- fails on the first call, succeeds on every call after. -class FlakyHandler { - fn handle(req: Req) -> Resp { - let n = len(from f in FlakyMark select f); - insert FlakyMark { n: 1 }; - if n == 0 { return server_error(); } - return ok_json("{\"ok\":true}"); - } -} - --- second reviewer follow-up: the SAME fails-once handler and table, --- separate from FlakyMark/FlakyHandler above so the sequential leg --- (18d) can't consume the one-time failure this leg (18e) needs -- but --- this time hit by two GENUINELY concurrent duplicates, to pin that --- neither ever receives a replayed 5xx from the other's ephemeral row. -@table(name: "flaky_marks2") -class FlakyMark2 { - n: Int -} - -class FlakyHandler2 { - fn handle(req: Req) -> Resp { - let n = len(from f in FlakyMark2 select f); - insert FlakyMark2 { n: 1 }; - if n == 0 { return server_error(); } - return ok_json("{\"ok\":true}"); - } -} - -class FlakyCount2 { - fn handle(req: Req) -> Resp { - let n = len(from f in FlakyMark2 select f); - return ok_json("{\"count\":${n}}"); - } -} - --- coordinator follow-up: ephemeral (4xx/5xx) rows must not accumulate. --- Always fails, so every attempt against the SAME idempotency key is --- its own ephemeral miss -- never durable, never a hit for the next one. -class AlwaysFailHandler { - fn handle(req: Req) -> Resp { - return server_error(); - } -} - -class IdemKeyCount { - fn handle(req: Req) -> Resp { - let n = len(from k in IdempotencyKey select k); - return ok_json("{\"count\":${n}}"); - } -} - --- reviewer finding 1: the README's own documented registration --- (`key_header: "Idempotency-Key"`, capitalised) against a wire header a --- real client also sends capitalised. Its own table/handler so the exec --- count is never confused with SlowHandler's. -@table(name: "casecheck_marks") -class CaseCheckMark { - n: Int -} - -class CaseCheckHandler { - fn handle(req: Req) -> Resp { - insert CaseCheckMark { n: 1 }; - let n = len(from c in CaseCheckMark select c); - return ok_json("{\"exec\":${n}}"); - } -} - -class CaseCheckCount { - fn handle(req: Req) -> Resp { - let n = len(from c in CaseCheckMark select c); - return ok_json("{\"count\":${n}}"); - } -} - --- reviewer finding 3: include_body:false must still scope the digest by --- method+path -- two distinct routes sharing one Idempotency-Key must --- never let the second replay the first's response. -class PathAHandler { - fn handle(req: Req) -> Resp { - return ok_json("{\"route\":\"a\"}"); - } -} - -class PathBHandler { - fn handle(req: Req) -> Resp { - return ok_json("{\"route\":\"b\"}"); - } -} - -fn build_app(slot: actor PoolMsg) -> App { - let app = App { middleware: [], routes: [] }; - let p = Pool { actors: [PoolSlot { a: slot }] }; - app.post("/create", Idempotent { key_header: "idempotency-key", pool: p, inner: SlowHandler {} }); - app.post("/flaky", Idempotent { key_header: "idempotency-key", pool: p, inner: FlakyHandler {} }); - app.post("/flaky2", Idempotent { key_header: "idempotency-key", pool: p, inner: FlakyHandler2 {} }); - app.post("/alwaysfail", Idempotent { key_header: "idempotency-key", pool: p, inner: AlwaysFailHandler {} }); - app.post("/casecheck", Idempotent { key_header: "Idempotency-Key", pool: p, inner: CaseCheckHandler {} }); - app.post("/patha", Idempotent { key_header: "idempotency-key", pool: p, inner: PathAHandler {}, include_body: false }); - app.post("/pathb", Idempotent { key_header: "idempotency-key", pool: p, inner: PathBHandler {}, include_body: false }); - app.get("/execs", ExecCount {}); - app.get("/flaky2count", FlakyCount2 {}); - app.get("/idemkeycount", IdemKeyCount {}); - app.get("/casecheckcount", CaseCheckCount {}); - return app; -} - -class Conn { fd: net.Conn } - -class ConnWorker { - slot: actor PoolMsg - fn receive(msg: Conn) { - let app = build_app(self.slot); - app.handle_conn(msg.fd, 5000, 5000); - } -} - -fn main(args: multi Text) -> Int { - if len(args) < 1 { - print_err("usage: idempotent_check "); - return 2; - } - let port = parse_int(args[0]); - if port == nil { print_err("bad port"); return 2; } - let ka: actor PoolMsg = spawn KeyActor {}; - let srv = net.listen("127.0.0.1", port); - print("listening on 127.0.0.1:${port}"); - while true { - if env.stopping() { net.close(srv); return 0; } - let c = net.accept_dl(srv, 250); - if c != nil { - let w: actor Conn = spawn ConnWorker { slot: ka }; - send(w, Conn { fd: c }); - } - } -} -WOEOF - -if ip_out="$("$WOC" --emit "$IP" -o "$IP/idempotent_check.wob" 2>&1)"; then - ok "idempotent: compiles (actor-run handler, digest, pool_begin)" - - IPORT=$((PORT + 2)) - IDATA="$W/idempotent-data"; mkdir -p "$IDATA" - printf '\n===== idempotent check — port %s =====\n' "$IPORT" >>"$SRVLOG" - LEGFROM=$(( $(wc -l < "$SRVLOG") + 1 )) - WO_DATA="$IDATA" "$WOVM" "$IP/idempotent_check.wob" "$IPORT" >>"$SRVLOG" 2>&1 & - SRV=$! - iwait_listen() { - for _ in $(seq 1 40); do - tail -n "+$LEGFROM" "$SRVLOG" 2>/dev/null | grep -q listening && return - sleep 0.1 - done - } - iwait_listen - - ipost() { # key body outfile -> prints STATUS, leaves the body in outfile - curl -s -o "$3" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: $1" -H "Content-Type: text/plain" \ - --data-binary "$2" "http://127.0.0.1:$IPORT/create" - } - iexecs() { # -> the ExecMark row count - curl -s --max-time 5 -H "Host: a" "http://127.0.0.1:$IPORT/execs" \ - | grep -o '"count":[0-9]*' | cut -d: -f2 - } - - # ---- 18a. gate leg: replay is exact (brief step 8) ---------------------- - s1="$(ipost leg8-key hello "$W/i8a.body")" - s2="$(ipost leg8-key hello "$W/i8b.body")" - [[ "$s1" == "200" && "$s2" == "200" ]] \ - && ok "idempotent: same key + same body both answer 200" \ - || bad "idempotent-replay-status" "s1=$s1 s2=$s2" - if cmp -s "$W/i8a.body" "$W/i8b.body"; then - ok "idempotent: replay is byte-identical" - else - bad "idempotent-replay-bytes" "$(cat "$W/i8a.body") != $(cat "$W/i8b.body")" - fi - ec="$(iexecs)" - [[ "$ec" == "1" ]] \ - && ok "idempotent: handler ran exactly once (ExecMark row count = 1)" \ - || bad "idempotent-replay-execs" "ExecMark count=$ec want 1" - - # ---- 18b. gate leg: digest mismatch is 422 (brief step 9) --------------- - m1="$(ipost leg9-key bodyA "$W/i9a.body")" - m2="$(ipost leg9-key bodyB "$W/i9b.body")" - [[ "$m1" == "200" && "$m2" == "422" ]] \ - && ok "idempotent: same key + different body is 422, not 200" \ - || bad "idempotent-mismatch-status" "m1=$m1 m2=$m2" - if ! cmp -s "$W/i9a.body" "$W/i9b.body"; then - ok "idempotent: 422 body is the refusal, not the other request's response" - else - bad "idempotent-mismatch-bytes" "422 body equals the first request's stored response" - fi - ec="$(iexecs)" - [[ "$ec" == "2" ]] \ - && ok "idempotent: the refused request never ran the handler (ExecMark row count = 2)" \ - || bad "idempotent-mismatch-execs" "ExecMark count=$ec want 2" - - # ---- 18c. gate leg: concurrent duplicates (brief step 10) --------------- - # Two backgrounded curl clients, launched together, hitting the SAME - # slow (500ms) handler through the SAME key -- a sequential version of - # this passes against the old before/after flow too and proves nothing. - t0=$(date +%s%3N) - ( s="$(ipost leg10-key samebody "$W/i10a.body")"; echo "$s" >"$W/i10a.status" ) & - cc1=$! - ( s="$(ipost leg10-key samebody "$W/i10b.body")"; echo "$s" >"$W/i10b.status" ) & - cc2=$! - wait "$cc1" "$cc2" - t1=$(date +%s%3N) - elapsed=$((t1 - t0)) - cs1="$(cat "$W/i10a.status")"; cs2="$(cat "$W/i10b.status")" - [[ "$cs1" == "200" && "$cs2" == "200" ]] \ - && ok "idempotent concurrency: both parallel duplicates answer 200" \ - || bad "idempotent-cc-status" "cs1=$cs1 cs2=$cs2" - if cmp -s "$W/i10a.body" "$W/i10b.body"; then - ok "idempotent concurrency: both clients got the same response body" - else - bad "idempotent-cc-bytes" "$(cat "$W/i10a.body") != $(cat "$W/i10b.body")" - fi - [[ "$elapsed" -lt 900 ]] \ - && ok "idempotent concurrency: genuinely overlapped (${elapsed}ms, serial would be ~1000ms+)" \ - || bad "idempotent-cc-elapsed" "${elapsed}ms" - ec="$(iexecs)" - [[ "$ec" == "3" ]] \ - && ok "idempotent concurrency: exactly one execution despite 2 parallel duplicates (ExecMark row count = 3)" \ - || bad "idempotent-cc-execs" "ExecMark count=$ec want 3" - - # ---- 18d. gate leg: a transient 5xx is never replayed (reviewer finding) -- - # The miss path must persist only a 2xx/3xx response. FlakyHandler fails - # on its first-ever call and succeeds on every call after; hit twice with - # the SAME idempotency key, the answer must be 500 then 200 -- caching the - # 500 would make every retry fail for the rest of the TTL (default 24h), - # a worse outcome than no idempotency at all. - f1="$(curl -s -o "$W/i11a.body" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: leg11-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT/flaky")" - f2="$(curl -s -o "$W/i11b.body" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: leg11-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT/flaky")" - [[ "$f1" == "500" && "$f2" == "200" ]] \ - && ok "idempotent: a transient 5xx is not replayed -- retry re-executes (500 then 200)" \ - || bad "idempotent-5xx-not-cached" "first=$f1 second=$f2 want 500 then 200" - - # ---- 18e. gate leg: a 5xx is never replayed to a CONCURRENT duplicate --- - # (coordinator follow-up on 18d's residual). Two backgrounded clients, - # launched together, same key, against a handler that fails only its - # first-ever invocation. Whichever message the actor's mailbox happens - # to process first gets that real failure; the second message must find - # the row ephemeral and re-run the handler itself -- never read back a - # replayed 500. Which of the two clients goes first is a race this test - # cannot pin, so it asserts the UNORDERED outcome instead: the statuses - # are exactly one 500 and one 200 (both requests genuinely executed -- - # FlakyMark2 count = 2). The pre-fix behavior (middleware-side delete, - # racing the actor) would show 500 and 500 with count = 1 whenever the - # duplicate is dequeued before the owner's delete lands -- deterministic - # either way, no ordering assumption needed. - ( s="$(curl -s -o "$W/i12a.body" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: leg12-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT/flaky2")"; echo "$s" >"$W/i12a.status" ) & - cf1=$! - ( s="$(curl -s -o "$W/i12b.body" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: leg12-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT/flaky2")"; echo "$s" >"$W/i12b.status" ) & - cf2=$! - wait "$cf1" "$cf2" - g1="$(cat "$W/i12a.status")"; g2="$(cat "$W/i12b.status")" - gsorted="$(printf '%s\n%s\n' "$g1" "$g2" | sort | tr '\n' ' ')" - [[ "$gsorted" == "200 500 " ]] \ - && ok "idempotent: concurrent duplicates never replay a 5xx (one 500, one 200)" \ - || bad "idempotent-5xx-concurrent" "g1=$g1 g2=$g2 want one 500 and one 200" - fc2="$(curl -s --max-time 5 -H "Host: a" "http://127.0.0.1:$IPORT/flaky2count" \ - | grep -o '"count":[0-9]*' | cut -d: -f2)" - [[ "$fc2" == "2" ]] \ - && ok "idempotent: both concurrent attempts genuinely executed (FlakyMark2 count = 2)" \ - || bad "idempotent-5xx-concurrent-execs" "FlakyMark2 count=$fc2 want 2" - - # ---- 18f. gate leg: ephemeral rows do not accumulate (coordinator follow-up) -- - # AlwaysFailHandler fails every time, so N attempts against the SAME - # idempotency key are N separate ephemeral misses -- never a durable - # row, never a hit for the next one. Before the fix each attempt left - # its own permanent, nonce-keyed row behind; after it, idempotent.wo - # deletes that row the instant it reads it back (safe: the nonce is - # never handed to anyone else, so nothing else could ever address that - # row anyway). The IdempotencyKey row count must return to its - # baseline after all N attempts, not grow by N. - idemkeycount() { - curl -s --max-time 5 -H "Host: a" "http://127.0.0.1:$IPORT/idemkeycount" \ - | grep -o '"count":[0-9]*' | cut -d: -f2 - } - ik_baseline="$(idemkeycount)" - IK_N=3 - for i in $(seq 1 $IK_N); do - curl -s -o /dev/null --max-time 5 -X POST -H "Host: a" \ - -H "Idempotency-Key: leg13-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT/alwaysfail" - done - ik_after="$(idemkeycount)" - [[ "$ik_after" == "$ik_baseline" ]] \ - && ok "idempotent: $IK_N ephemeral attempts leave no rows behind (IdempotencyKey count stays $ik_baseline)" \ - || bad "idempotent-ephemeral-leak" "baseline=$ik_baseline after $IK_N attempts=$ik_after" - - kill -TERM "$SRV" 2>/dev/null - istopped=1 - for _ in $(seq 1 30); do kill -0 "$SRV" 2>/dev/null || { istopped=0; break; }; sleep 0.1; done - [[ $istopped -eq 0 ]] && ok "idempotent: SIGTERM stops the server" || bad "idempotent-stop" "still running" - if [[ $istopped -eq 1 ]]; then - # §14/§17b's own pattern clears SRV here unconditionally, which is - # exactly how an orphan survives past this leg: the EXIT trap only - # kills a non-empty $SRV, so a still-running process that this loop - # gave up on would otherwise keep the port bound for the NEXT run of - # this whole script. Force it dead right here instead of trusting the - # trap -- the bad-verdict line above already told the reader SIGTERM - # alone did not work. - kill -9 "$SRV" 2>/dev/null - for _ in $(seq 1 20); do kill -0 "$SRV" 2>/dev/null || break; sleep 0.1; done - fi - SRV="" - - # ---- 18g/18h: reviewer findings 1 and 3, on a FRESH restart ------------- - # Same compiled binary, fresh WO_DATA, a new port -- not piled onto 18a-18f's - # already-loaded server. 18a-18f alone already spawn a dozen-plus per- - # connection actors on that one process; adding these two legs there measurably - # raised how often this run hit the pre-existing, out-of-scope C-runtime - # arena-allocator race (a SIGSEGV inside wo_str_new, confirmed by gdb - # backtrace -- unrelated to this file's own .wo logic; see the story's - # Outstanding notes). A fresh process for these two legs keeps them just as - # rigorous while giving that race far less to chew on. - IPORT2=$((IPORT + 50)) - IDATA2="$W/idempotent-data-2"; mkdir -p "$IDATA2" - printf '\n===== idempotent check — reviewer findings 1/3, port %s =====\n' "$IPORT2" >>"$SRVLOG" - LEGFROM=$(( $(wc -l < "$SRVLOG") + 1 )) - WO_DATA="$IDATA2" "$WOVM" "$IP/idempotent_check.wob" "$IPORT2" >>"$SRVLOG" 2>&1 & - SRV=$! - iwait_listen2() { - for _ in $(seq 1 40); do - tail -n "+$LEGFROM" "$SRVLOG" 2>/dev/null | grep -q listening && return - sleep 0.1 - done - } - iwait_listen2 - - # ---- 18g. gate leg: README's own documented registration, over the wire - # a real client actually sends (reviewer finding 1) ---------------------- - # key_header: "Idempotency-Key" (capitalised, exactly as the README told - # app authors to write it) against a wire header ALSO sent capitalised. - # parse.wo lowercases every header name on read, so a lookup on the - # unnormalised self.key_header always misses -- pre-fix this falls - # through to self.inner.handle(req) on EVERY call, so the second - # request is never deduplicated: the exec count is the load-bearing - # assertion, not the status (both calls answer 200 either way). - cc1="$(curl -s -o "$W/i14a.body" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: leg14-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT2/casecheck")" - cc2="$(curl -s -o "$W/i14b.body" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: leg14-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT2/casecheck")" - [[ "$cc1" == "200" && "$cc2" == "200" ]] \ - && ok "idempotent: capitalised key_header + capitalised wire header both answer 200" \ - || bad "idempotent-header-case-status" "cc1=$cc1 cc2=$cc2" - casecheckcount="$(curl -s --max-time 5 -H "Host: a" "http://127.0.0.1:$IPORT2/casecheckcount" \ - | grep -o '"count":[0-9]*' | cut -d: -f2)" - [[ "$casecheckcount" == "1" ]] \ - && ok "idempotent: README's documented capitalised key_header still dedupes (CaseCheckMark count = 1)" \ - || bad "idempotent-header-case-execs" "CaseCheckMark count=$casecheckcount want 1" - - # ---- 18h. gate leg: include_body:false must still scope by method+path - # (reviewer finding 3) ---------------------------------------------------- - # Same Idempotency-Key against two DIFFERENT routes, include_body:false on - # both. Pre-fix the digest is "" unconditionally when include_body is - # false, so the second route's request matches the first route's stored - # bare-key row on digest alone and replays route a's body under route b. - # Post-fix, method+path are always part of the digest, so this is the - # SAME "different request, same key" case leg 18b already pins for a body - # mismatch -- route a succeeds (200), route b is refused (422), and - # route b's body must be the refusal, never route a's replayed body. - d1="$(curl -s -o "$W/i15a.body" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: leg15-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT2/patha")" - d2="$(curl -s -o "$W/i15b.body" -w '%{http_code}' --max-time 5 -X POST \ - -H "Host: a" -H "Idempotency-Key: leg15-key" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$IPORT2/pathb")" - [[ "$d1" == "200" && "$d2" == "422" ]] \ - && ok "idempotent: include_body:false, same key on two different routes: first 200, second refused (422)" \ - || bad "idempotent-diffpath-status" "d1=$d1 d2=$d2 want 200 then 422" - if grep -q '"route":"a"' "$W/i15b.body"; then - bad "idempotent-diffpath-replay" "pathb's response replayed patha's body: $(cat "$W/i15b.body")" - else - ok "idempotent: include_body:false does not replay a different route's response across paths" - fi - - kill -TERM "$SRV" 2>/dev/null - istopped2=1 - for _ in $(seq 1 30); do kill -0 "$SRV" 2>/dev/null || { istopped2=0; break; }; sleep 0.1; done - [[ $istopped2 -eq 0 ]] && ok "idempotent (findings 1/3): SIGTERM stops the server" || bad "idempotent-stop-2" "still running" - if [[ $istopped2 -eq 1 ]]; then - kill -9 "$SRV" 2>/dev/null - for _ in $(seq 1 20); do kill -0 "$SRV" 2>/dev/null || break; sleep 0.1; done - fi - SRV="" -else - bad "idempotent-compile" "$(printf '%s' "$ip_out" | head -1)" -fi - -# ---- 19. porch-store task 5: pool saturation fails closed (503, no bypass) -- -# Same flattening trick as the earlier legs. WO_MAILBOX (runtime/src/vm.c, -# wo_mailbox_cap, default 1024) shrinks the runtime's per-actor mailbox cap -# so a handful of concurrent requests can actually exhaust it. A pool of -# ONE actor -- the only address a one-slot Pool's pool_select can ever -# return -- fed a handler that blocks it for SP_SLEEP_MS turns every -# genuinely-concurrent request into a race for that one mailbox's slots. -# Distinct idempotency keys per request rule out replay masking a request -# that never actually ran the handler. -# -# The runtime frees a reserved slot the instant a message is POPPED for -# delivery, not when its receive returns (wo_mbox_reserve/release), so -# with cap C exactly the first C+1 concurrent calls to the one busy actor -# ever get a slot -- one executing, C queued behind it -- and every later -# concurrent call finds the mailbox full and traps (WO_T_ACTOR), which -# idempotent.wo's own try/catch turns into 503. SP_SLEEP_MS only has to -# outlast the time it takes SP_N curl clients to all reach their `call`, -# comfortably true on localhost. -SP="$W/saturation-check" -cp -r "$ROOT/docs/examples/porch" "$SP" -rm -f "$SP/wo.toml" -rm -rf "$SP/target" -cat >"$SP/saturation_check_main.wo" <<'WOEOF' -use net -use env -use http -use router -use middleware -use time - -@table(name: "sat_execs") -class SatMark { - n: Int -} - --- Blocks the pool's one actor for a few seconds on every genuine --- (non-replay) execution -- the same shape as idempotent-check's --- SlowHandler (section 18), its own table so the two legs' counts can --- never be confused. -class SlowSatHandler { - fn handle(req: Req) -> Resp { - insert SatMark { n: 1 }; - time.sleep(3000); - return ok_json("{\"ok\":true}"); - } -} - -class SatExecCount { - fn handle(req: Req) -> Resp { - let n = len(from e in SatMark select e); - return ok_json("{\"count\":${n}}"); - } -} - -fn build_app(slot: actor PoolMsg) -> App { - let app = App { middleware: [], routes: [] }; - let p = Pool { actors: [PoolSlot { a: slot }] }; - app.post("/slow", Idempotent { key_header: "idempotency-key", pool: p, inner: SlowSatHandler {} }); - app.get("/execs", SatExecCount {}); - return app; -} - -class Conn { fd: net.Conn } - -class ConnWorker { - slot: actor PoolMsg - fn receive(msg: Conn) { - let app = build_app(self.slot); - app.handle_conn(msg.fd, 8000, 8000); - } -} - -fn main(args: multi Text) -> Int { - if len(args) < 1 { - print_err("usage: saturation_check "); - return 2; - } - let port = parse_int(args[0]); - if port == nil { print_err("bad port"); return 2; } - let ka: actor PoolMsg = spawn KeyActor {}; - let srv = net.listen("127.0.0.1", port); - print("listening on 127.0.0.1:${port}"); - while true { - if env.stopping() { net.close(srv); return 0; } - let c = net.accept_dl(srv, 250); - if c != nil { - let w: actor Conn = spawn ConnWorker { slot: ka }; - send(w, Conn { fd: c }); - } - } -} -WOEOF - -if sp_out="$("$WOC" --emit "$SP" -o "$SP/saturation_check.wob" 2>&1)"; then - ok "saturation: compiles (one-actor pool, slow in-actor handler)" - - SPORT=$((PORT + 3)) - SPDATA="$W/saturation-data"; mkdir -p "$SPDATA" - SPSTATUS="$W/saturation-status"; mkdir -p "$SPSTATUS" - SP_CAP=2 - SP_N=15 - printf '\n===== saturation check — port %s (WO_MAILBOX=%s) =====\n' "$SPORT" "$SP_CAP" >>"$SRVLOG" - LEGFROM=$(( $(wc -l < "$SRVLOG") + 1 )) - WO_DATA="$SPDATA" WO_MAILBOX="$SP_CAP" "$WOVM" "$SP/saturation_check.wob" "$SPORT" >>"$SRVLOG" 2>&1 & - SRV=$! - spwait_listen() { - for _ in $(seq 1 40); do - tail -n "+$LEGFROM" "$SRVLOG" 2>/dev/null | grep -q listening && return - sleep 0.1 - done - } - spwait_listen - - # SP_N genuinely-parallel duplicates, each its own idempotency key, all - # against the SAME (one-actor) pool -- a sequential version proves nothing, - # same reasoning as every other concurrency leg in this file. - sp_pids=() - for i in $(seq 1 $SP_N); do - ( st="$(curl -s -D "$SPSTATUS/$i.hdr" -o "$SPSTATUS/$i.body" -w '%{http_code}' --max-time 15 -X POST \ - -H "Host: a" -H "Idempotency-Key: sat-key-$i" -H "Content-Type: text/plain" \ - --data-binary "x" "http://127.0.0.1:$SPORT/slow")" - echo "$st" >"$SPSTATUS/$i.status" ) & - sp_pids+=("$!") - done - for p in "${sp_pids[@]}"; do wait "$p"; done - - sp_200=0 - sp_503=0 - sp_other=0 - sp_one503="" - for i in $(seq 1 $SP_N); do - st="$(cat "$SPSTATUS/$i.status" 2>/dev/null)" - case "$st" in - 200) sp_200=$((sp_200 + 1)) ;; - 503) sp_503=$((sp_503 + 1)); sp_one503="$i" ;; - *) sp_other=$((sp_other + 1)) ;; - esac - done - sp_want_ok=$((SP_CAP + 1)) - sp_want_bad=$((SP_N - sp_want_ok)) - [[ "$sp_other" -eq 0 ]] \ - && ok "saturation: every one of $SP_N requests answered 200 or 503, nothing else" \ - || bad "saturation-codes" "$sp_other requests answered neither (200=$sp_200 503=$sp_503)" - [[ "$sp_200" -eq "$sp_want_ok" && "$sp_503" -eq "$sp_want_bad" ]] \ - && ok "saturation: exactly $sp_want_ok served (1 running + $SP_CAP queued), $sp_want_bad overflow answer 503" \ - || bad "saturation-threshold" "200=$sp_200 503=$sp_503 want 200=$sp_want_ok 503=$sp_want_bad" - - sp_execs="$(curl -s --max-time 5 -H "Host: a" "http://127.0.0.1:$SPORT/execs" \ - | grep -o '"count":[0-9]*' | cut -d: -f2)" - [[ "$sp_execs" == "$sp_200" ]] \ - && ok "saturation: handler ran exactly once per 200 (SatMark count=$sp_execs) -- no overflow request slipped through uncounted" \ - || bad "saturation-execs" "SatMark count=$sp_execs want $sp_200 (== the 200 count)" - - if [[ -n "$sp_one503" ]]; then - grep -qi '^retry-after:' "$SPSTATUS/$sp_one503.hdr" \ - && ok "saturation 503 carries Retry-After" \ - || bad "saturation-503-retry-after" "$(head -1 "$SPSTATUS/$sp_one503.hdr")" - grep -q 'idempotency store saturated' "$SPSTATUS/$sp_one503.body" \ - && ok "saturation 503 names the real cause (idempotency store saturated), not a generic failure" \ - || bad "saturation-503-body" "$(cat "$SPSTATUS/$sp_one503.body")" - else - bad "saturation-503-missing" "no 503 observed among $SP_N requests -- cannot verify overflow shape" - fi - - # Teardown is deliberately NOT asserted pass/fail here (unlike the earlier - # legs' own SIGTERM checks): this leg's subject is saturation, not graceful - # shutdown -- already proven in §14 and (usually) §17b/§18. This exact - # workload -- many concurrent call()-parked callers against one busy actor - # doing real per-request table I/O -- is the sharpest known trigger for a - # pre-existing runtime defect (see the story's Outstanding notes): main() - # can return cleanly while the OS process itself hangs. Failing this leg - # over that already-documented, out-of-scope defect would be exactly the - # kind of flaky check that erodes trust in every other leg in this file, so - # it force-kills instead of asserting graceful-vs-forced. - kill -TERM "$SRV" 2>/dev/null - spstopped=1 - for _ in $(seq 1 30); do kill -0 "$SRV" 2>/dev/null || { spstopped=0; break; }; sleep 0.1; done - if [[ $spstopped -eq 1 ]]; then - kill -9 "$SRV" 2>/dev/null - for _ in $(seq 1 20); do kill -0 "$SRV" 2>/dev/null || break; sleep 0.1; done - fi - note "saturation: server torn down (graceful SIGTERM, or kill -9 on the known actor-pool hang)" - SRV="" -else - bad "saturation-compile" "$(printf '%s' "$sp_out" | head -1)" -fi echo printf 'web-app-accept: %d checks, %d failures\n' "$((pass + fail))" "$fail"