diff --git a/docs/stories/00-status.md b/docs/stories/00-status.md index f494061..836ec06 100644 --- a/docs/stories/00-status.md +++ b/docs/stories/00-status.md @@ -1195,9 +1195,9 @@ the language arc as v1 history. ### ▸ porch — the web framework track New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md). -Supersedes language iteration 39, now a pointer. **Stories 2–6 are `ready` +Supersedes language iteration 39, now a pointer. **Stories 2–7 are `ready` (brainstormed 2026-09-06, forks locked, validated against -`.dev/reference/fiber`); 7–8 remain `refine`.** Ordered by dependency; the +`.dev/reference/fiber`); 8 remains `refine`.** Ordered by dependency; the first slice is deliberately the cheapest so the store pattern and gate shape are proven before the runtime and `Resp` are touched. @@ -1209,7 +1209,7 @@ proven before the runtime and `Resp` are touched. | 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 | | 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) | | 6 | [Streaming core](porch/06-streaming-core.md) | ✅ **`ready` 2026-09-06** — riskiest/highest-leverage; **re-scoped to outbound only**. Three decisions: separate `StreamHandler`/`BodyProducer` parallel path (the `Resp` path untouched → existing responses byte-identical); streaming routes **opt out** of the after-chain, framework **refuses at registration** to combine with header-mutating middleware (loud, never silent), handlers stamp headers via a `security_headers()` helper; chunked **REQUEST** bodies **split into their own future iteration** (parse.wo refusal stays). No language enhancement; rides the fiber loop, not the actor pool (not lang-41-exposed) | -| 7 | [SSE + compression](porch/07-sse-and-compression.md) | ⬜ after 6. SSE fits the actor/fiber model unusually well; compression carries a real fork — pure-`.wo` DEFLATE (now expressible after iteration 36's bit operators) vs a C builtin. CRC32 finally gets its consumer | +| 7 | [SSE + compression](porch/07-sse-and-compression.md) | ✅ **`ready` 2026-09-06** — after 6 (+ 5 for q-ranking/comma-join Vary; NOT 2). Five decisions: refuse an incoherent heartbeat/`idle_ms` pair at construction; **codec = two C builtins `deflate`+`crc32`** (perf over pure-`.wo`; hand-rolled, no zlib dep; gzip framing in `.wo`) — the track's **second language dependency** after iteration 2; ETag over uncompressed bytes + `Vary`; `Last-Event-ID` explicitly unsupported (not silently ignored); Vary via comma-join. Codec is pure compute — not lang-41-exposed | | 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip | --- diff --git a/docs/stories/porch/07-sse-and-compression.md b/docs/stories/porch/07-sse-and-compression.md index 5616446..54fa303 100644 --- a/docs/stories/porch/07-sse-and-compression.md +++ b/docs/stories/porch/07-sse-and-compression.md @@ -2,46 +2,81 @@ track: porch iteration: "7" status: pending -readiness: refine +readiness: ready --- # porch 7 — server-sent events and compression > Part of [Story — `porch`, the writeonce web framework](00-story.md). -> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3. -> Both halves need [iteration 6](06-streaming-core.md); neither is possible -> before it. +> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3, +> re-checked 2026-09-06 against `.dev/reference/fiber` (v3, `3ca9a9d`). +> Both halves need [iteration 6](06-streaming-core.md); the compression half also +> needs [iteration 5](05-routing-response-ergonomics.md)'s q-value ranking and +> comma-join `Vary`, and **two new language-track builtins** (decision 2). +> Not dependent on iteration 2, and not exposed to the lang-41 hang. ## Goals - **SSE, which fits this runtime unusually well.** A room actor already has the - fan-out shape a live feed needs, and a parked fiber per subscriber costs - almost nothing on the shard model — so the awkward part of SSE in most - frameworks (holding thousands of idle connections) is the part writeonce - already solved with iteration 35's idle deadlines and fiber-per-connection. - Fiber ships `Retry`, `HeartbeatInterval` and `OnClose`; all three matter, - because a proxy will silently drop an idle event stream. + fan-out shape a live feed needs, and a parked fiber per subscriber costs almost + nothing on the shard model — so the awkward part of SSE in most frameworks + (holding thousands of idle connections) is the part writeonce already solved + with iteration 35's idle deadlines and fiber-per-connection. Fiber ships + `Retry`, `HeartbeatInterval` and `OnClose`; all three matter, because a proxy + will silently drop an idle event stream. - **gzip/deflate, decided honestly.** Iteration 36 landed the bitwise operators, - so a pure-`.wo` DEFLATE is now *expressible* — the question is whether it - should be. A C builtin is faster and smaller to write; a `.wo` implementation - keeps the runtime doctrine intact and proves the language can do real - bit-level work. Fork 2 decides, and the answer should turn on whether anything - else will ever want zlib. -- **Negotiate, never assume.** Compress only when the client said it accepts the - coding, only above a size threshold, and never for content that is already - compressed — a gzipped JPEG is bigger than the JPEG. + so a pure-`.wo` DEFLATE is now expressible — but the brainstorm chose C + builtins for the hot path (decision 2). Negotiate, never assume: compress only + when the client accepts the coding, only above a size threshold, and never for + content that is already compressed. + +## Decisions locked (brainstorm 2026-09-06) + +1. **An incoherent heartbeat/idle pair is refused at construction.** A heartbeat + interval at or above the connection's `idle_ms` evicts the very client the + heartbeat exists to keep — so the SSE endpoint refuses that pair when it is + built, with a message naming both values. A misconfiguration is a startup + error, never a flaky-network mystery — the same refuse-the-unconstructible + discipline story 6 used for streaming plus header-mutating middleware. +2. **The codec is two C builtins: `deflate` (raw) and `crc32`; gzip framing is + assembled in `.wo`.** This is the iteration's language-track work, called out + like iteration 2's phase A. The brainstorm chose performance over the + pure-`.wo` option: LZ77 match-finding and a per-byte CRC table are inner-loop + heavy and belong in C. They live in the crypto-family builtin table beside + `sha256`, hand-rolled with **no zlib dependency** (the `sha256` precedent). + `deflate` returns the raw stream; `crc32` returns the 32-bit checksum; the + gzip container (10-byte header, deflate body, little-endian CRC32 and ISIZE + trailer) is built in `.wo` with the iteration-36 bit operators. Decompression + (`inflate`) is not built — request-body decompression is out of scope. +3. **The ETag is computed over the uncompressed bytes, before compression, and + paired with `Vary: Accept-Encoding`.** `etag_for`/`with_etag` run on the + handler's uncompressed body as today; the middleware compresses afterward. The + ETag identifies the resource and is stable across encodings; `Vary` keys + caches on the encoding so a gzipped body never reaches an identity-only client; + a 304 carries no body, so the encoding is moot on a conditional hit. The + per-representation/weak-ETag alternative is more RFC-strict but makes the + middleware rewrite the handler's tag and is easy to get wrong — not chosen. +4. **`Last-Event-ID` resumption is not supported this iteration, and says so.** + Real resumption needs a per-subscriber replay buffer, which no workload asks + for yet. The client's `Last-Event-ID` is not silently ignored — that would let + a client believe it resumed when it did not; the endpoint states plainly that + it does not resume. +5. **`Vary` accumulation uses iteration 5's comma-join, not iteration 2.** This + corrects the draft: story 5 decided `Vary` accumulates by comma-joining in the + existing header map, so the compression middleware composes with other `Vary` + contributors without iteration 2's repeated-header work. ## Phases ### Phase A — SSE framing and lifecycle - The event framing (`data:`, `event:`, `id:`, `retry:`), the double-newline - terminator, and the `text/event-stream` content type with caching disabled. -- Heartbeats, because an idle stream through a proxy dies quietly. Interacts - directly with iteration 35's `idle_ms` — a heartbeat interval longer than the - idle deadline evicts the client the heartbeat exists to keep. -- Disconnect detection and cleanup: a write to a gone client must free the - fiber, the actor subscription and the fd, on every path. + terminator, and the `text/event-stream` content type with caching disabled, + emitted through iteration 6's streaming writer. +- Heartbeats, with the construction-time refusal from decision 1 guarding the + interaction with iteration 35's `idle_ms`. +- Disconnect detection and cleanup: a write to a gone client must free the fiber, + the actor subscription and the fd, on every path. - Verify: a client receives ordered events; a heartbeat keeps an otherwise idle stream alive past `idle_ms`; a disconnect releases everything (fd count flat). @@ -49,42 +84,41 @@ readiness: refine - A live feed in a sample fed by an actor, so the fan-out path is real rather than a loop in one handler. -- `Last-Event-ID` resumption, or an explicit statement that it is not supported - — silently ignoring it means clients think they resumed when they did not. +- `Last-Event-ID` handling per decision 4: not supported, stated explicitly. - Verify: N concurrent subscribers all receive an event published once; fds and memory flat across a churn of connect/disconnect. -### Phase C — the compression codec +### Phase C — the compression codec (language-track work) -- Implement or bind the codec per fork 2, with the framing gzip requires - (header, deflate stream, CRC32 and length trailer). Note CRC32 does not exist - yet — the crypto row lists it as waiting for a consumer, and this is that - consumer. -- Correctness against a reference decompressor is the acceptance bar, on - awkward input: empty, highly repetitive, incompressible, and larger than any - internal buffer. -- Verify: `gzip -d` reproduces the input byte-exactly for every case. +- Add the `deflate` and `crc32` builtins (decision 2), hand-rolled in C with no + external dependency, documented in the builtin-surface contract and error + catalog in the same change. CRC32 is the consumer the crypto row has listed as + waiting. +- Assemble the gzip container in `.wo`: header, deflate stream, CRC32 and ISIZE + trailer. +- Correctness against a reference decompressor is the acceptance bar, on awkward + input: empty, highly repetitive, incompressible, and larger than any internal + buffer. +- Verify: `oop-e2e` green for the builtins; `gzip -d` reproduces the input + byte-exactly for every case. ### Phase D — the compression middleware -- `Accept-Encoding` negotiation reusing the q-value ranking iteration - [5](05-routing-response-ergonomics.md) added, a minimum-size threshold, and a - content-type skip list. -- `Vary: Accept-Encoding` on anything compressed, or every cache in front will - serve gzipped bytes to a client that cannot read them. This needs iteration - 2's repeated-header work to accumulate correctly with other `Vary` - contributions. -- Composition with chunked streaming: compress then chunk, and the ETag question - — an ETag computed over compressed bytes is a different entity than the same - resource uncompressed. -- Verify: a compressed response decompresses to the original; `Vary` is present; - no double-compression; an already-compressed content type is skipped. +- `Accept-Encoding` negotiation reusing iteration 5's q-value ranking, a + minimum-size threshold, and a content-type skip list. +- `Vary: Accept-Encoding` on anything compressed, accumulated by comma-join + (decision 5) so it coexists with other `Vary` contributions. +- Composition with chunked streaming: compress then chunk; the ETag rule from + decision 3. +- Verify: a compressed response decompresses to the original; `Vary` is present + and coexists; no double-compression; an already-compressed content type is + skipped. ### Phase E — the gate and the ledger -- Both serving gates plus an ASan leg for the codec, which is the part most - likely to leak or over-read. -- Retire the ledger's SSE and compression rows. +- Both serving gates plus an ASan leg for the codec, the part most likely to leak + or over-read. +- Retire the ledger's SSE and compression rows; record the CRC32 consumer. - Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean. ## Acceptance Criteria @@ -93,53 +127,49 @@ readiness: refine published, **then** the client receives them in order with correct framing. - **Given** an idle SSE stream and a heartbeat interval shorter than `idle_ms`, **when** it idles past the deadline, **then** it stays open. -- **Given** a heartbeat interval *longer* than `idle_ms`, **when** the stream - idles, **then** the misconfiguration is evident rather than mysterious — the - interaction is documented and, ideally, refused at construction. -- **Given** a client disconnecting mid-stream, **when** the next publish - occurs, **then** the write failure frees the fiber, the subscription and the - fd; fd count returns to baseline. +- **Given** a heartbeat interval at or above `idle_ms`, **when** the endpoint is + constructed, **then** it is refused with a message naming both values. +- **Given** a client disconnecting mid-stream, **when** the next publish occurs, + **then** the write failure frees the fiber, the subscription and the fd; fd + count returns to baseline. - **Given** N concurrent subscribers, **when** one event is published, **then** all N receive it and memory does not grow per event. -- **Given** any input including empty, repetitive and incompressible, **when** - it is compressed, **then** a reference `gzip -d` reproduces it byte-exactly. +- **Given** any input including empty, repetitive and incompressible, **when** it + is compressed, **then** a reference `gzip -d` reproduces it byte-exactly. - **Given** a client that did not send `Accept-Encoding`, **when** it requests a compressible resource, **then** the response is uncompressed. - **Given** a compressed response, **when** it leaves, **then** `Vary: Accept-Encoding` is set and coexists with any other `Vary` contribution. - **Given** an already-compressed content type, **when** it is served, **then** it is not compressed again. +- **Given** a conditional request for a compressible resource, **when** its + `If-None-Match` matches the uncompressed-body ETag, **then** a 304 is returned + regardless of the client's `Accept-Encoding`. ## Out Of Scope - **Brotli and zstd.** One codec, proven, before a second. gzip is what every client accepts. -- **Request-body decompression.** A compressed *upload* is a separate surface - with its own decompression-bomb risk, and no workload asks yet. -- **WebSockets as an SSE alternative.** Already shipped and a different tool; - SSE is the one-way, proxy-friendly, reconnect-by-default option. +- **Request-body decompression (`inflate`).** A compressed *upload* is a separate + surface with its own decompression-bomb risk, and no workload asks yet — so the + codec builtins are compress-only. +- **`Last-Event-ID` resumption** — decision 4; needs a replay buffer no workload + wants yet. +- **WebSockets as an SSE alternative.** Already shipped and a different tool; SSE + is the one-way, proxy-friendly, reconnect-by-default option. - **Compressing static files at rest.** Fiber's static has `Compress`; precompressed-file serving belongs with iteration [8](08-static-and-lifecycle.md). -- **A general-purpose zlib library surface.** Whatever lands is what the - middleware needs. If a second consumer appears, it can argue for a library. +- **A general-purpose zlib library surface.** The builtins are what the middleware + needs (`deflate`, `crc32`); if a second consumer appears it can argue for + `inflate` and a library. ## Info -Forks the spec must settle: - -1. **Heartbeat versus idle deadline.** These two mechanisms can silently fight, - and the failure looks like a flaky network. Decide whether the framework - refuses an incoherent pair at construction — leaning yes, because a - configuration that cannot work should not be constructible. -2. **`.wo` DEFLATE or a C builtin?** The honest tiebreaker is whether anything - else ever wants zlib. If compression is the only consumer forever, a `.wo` - implementation keeps the runtime small and is a genuine demonstration that - iteration 36's bit operators earned their place. If a second consumer is - plausible (precompressed assets, a WAL codec, an archive format), the builtin - wins. CRC32 comes along either way. -3. **ETag over compressed or uncompressed bytes?** `etag_for` exists and is used - with `with_etag` for 304s. Compressing after the ETag is computed keeps the - entity identity stable across encodings, which is almost certainly right — - but it must be decided, because getting it wrong serves the wrong body for a - conditional request. +This is the second porch iteration with a language dependency (after iteration +2): two builtins, `deflate` and `crc32`, chosen over the pure-`.wo` path for +hot-path performance. Everything else is pure `.wo` on iteration 6's streaming +writer and iteration 5's negotiation. The codec is pure compute — no actors, no +per-key pool — so, like the rest of the track since iteration 1, it is not +exposed to the lang-41 hang. The risk concentrates in the C codec (ASan leg in +phase E) and in the SSE lifecycle's fd/subscription cleanup.