docs(porch-sse): brainstorm story 7 (SSE + compression) to ready

- five decisions: refuse incoherent heartbeat/idle_ms pair at construction;
  codec = two C builtins deflate+crc32 (perf over pure-.wo; hand-rolled, no
  zlib dep; gzip framing in .wo); ETag over uncompressed bytes + Vary;
  Last-Event-ID explicitly unsupported (not silently ignored); Vary via
  comma-join
- language enhancement: YES, two builtins -- the track's SECOND language
  dependency after iteration 2's random_bytes. CRC32 finally gets its
  consumer; inflate deliberately not built (request-body decompression OOS)
- corrected stale dependency: Vary uses iteration 5's comma-join, so story 7
  depends on 6 + 5, NOT 2; codec is pure compute, not lang-41-exposed
- confirmed CRC32 absent + iteration 36 bit operators landed (pure-.wo was
  viable, traded for hot-path speed)
- validated against .dev/reference/fiber. Board synced

(cherry picked from commit 07f53574dd90f502235b79d4920eab3d684c8b77)
This commit is contained in:
shoney.arickathil 2026-09-06 17:41:03 +02:00
parent 641703903c
commit 4d3e4261e1
2 changed files with 114 additions and 84 deletions

View file

@ -1195,9 +1195,9 @@ the language arc as v1 history.
### ▸ porch — the web framework track ### ▸ porch — the web framework track
New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md). New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md).
Supersedes language iteration 39, now a pointer. **Stories 2–6 are `ready` Supersedes language iteration 39, now a pointer. **Stories 2–7 are `ready`
(brainstormed 2026-09-06, forks locked, validated against (brainstormed 2026-09-06, forks locked, validated against
`.dev/reference/fiber`); 7–8 remain `refine`.** Ordered by dependency; the `.dev/reference/fiber`); 8 remains `refine`.** Ordered by dependency; the
first slice is deliberately the cheapest so the store pattern and gate shape are first slice is deliberately the cheapest so the store pattern and gate shape are
proven before the runtime and `Resp` are touched. proven before the runtime and `Resp` are touched.
@ -1209,7 +1209,7 @@ proven before the runtime and `Resp` are touched.
| 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 | | 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 |
| 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) | | 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) |
| 6 | [Streaming core](porch/06-streaming-core.md) | ✅ **`ready` 2026-09-06** — riskiest/highest-leverage; **re-scoped to outbound only**. Three decisions: separate `StreamHandler`/`BodyProducer` parallel path (the `Resp` path untouched → existing responses byte-identical); streaming routes **opt out** of the after-chain, framework **refuses at registration** to combine with header-mutating middleware (loud, never silent), handlers stamp headers via a `security_headers()` helper; chunked **REQUEST** bodies **split into their own future iteration** (parse.wo refusal stays). No language enhancement; rides the fiber loop, not the actor pool (not lang-41-exposed) | | 6 | [Streaming core](porch/06-streaming-core.md) | ✅ **`ready` 2026-09-06** — riskiest/highest-leverage; **re-scoped to outbound only**. Three decisions: separate `StreamHandler`/`BodyProducer` parallel path (the `Resp` path untouched → existing responses byte-identical); streaming routes **opt out** of the after-chain, framework **refuses at registration** to combine with header-mutating middleware (loud, never silent), handlers stamp headers via a `security_headers()` helper; chunked **REQUEST** bodies **split into their own future iteration** (parse.wo refusal stays). No language enhancement; rides the fiber loop, not the actor pool (not lang-41-exposed) |
| 7 | [SSE + compression](porch/07-sse-and-compression.md) | ⬜ after 6. SSE fits the actor/fiber model unusually well; compression carries a real fork — pure-`.wo` DEFLATE (now expressible after iteration 36's bit operators) vs a C builtin. CRC32 finally gets its consumer | | 7 | [SSE + compression](porch/07-sse-and-compression.md) | ✅ **`ready` 2026-09-06** — after 6 (+ 5 for q-ranking/comma-join Vary; NOT 2). Five decisions: refuse an incoherent heartbeat/`idle_ms` pair at construction; **codec = two C builtins `deflate`+`crc32`** (perf over pure-`.wo`; hand-rolled, no zlib dep; gzip framing in `.wo`) — the track's **second language dependency** after iteration 2; ETag over uncompressed bytes + `Vary`; `Last-Event-ID` explicitly unsupported (not silently ignored); Vary via comma-join. Codec is pure compute — not lang-41-exposed |
| 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip | | 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip |
--- ---

View file

@ -2,46 +2,81 @@
track: porch track: porch
iteration: "7" iteration: "7"
status: pending status: pending
readiness: refine readiness: ready
--- ---
# porch 7 — server-sent events and compression # porch 7 — server-sent events and compression
> Part of [Story — `porch`, the writeonce web framework](00-story.md). > Part of [Story — `porch`, the writeonce web framework](00-story.md).
> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3. > Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3,
> Both halves need [iteration 6](06-streaming-core.md); neither is possible > re-checked 2026-09-06 against `.dev/reference/fiber` (v3, `3ca9a9d`).
> before it. > Both halves need [iteration 6](06-streaming-core.md); the compression half also
> needs [iteration 5](05-routing-response-ergonomics.md)'s q-value ranking and
> comma-join `Vary`, and **two new language-track builtins** (decision 2).
> Not dependent on iteration 2, and not exposed to the lang-41 hang.
## Goals ## Goals
- **SSE, which fits this runtime unusually well.** A room actor already has the - **SSE, which fits this runtime unusually well.** A room actor already has the
fan-out shape a live feed needs, and a parked fiber per subscriber costs fan-out shape a live feed needs, and a parked fiber per subscriber costs almost
almost nothing on the shard model — so the awkward part of SSE in most nothing on the shard model — so the awkward part of SSE in most frameworks
frameworks (holding thousands of idle connections) is the part writeonce (holding thousands of idle connections) is the part writeonce already solved
already solved with iteration 35's idle deadlines and fiber-per-connection. with iteration 35's idle deadlines and fiber-per-connection. Fiber ships
Fiber ships `Retry`, `HeartbeatInterval` and `OnClose`; all three matter, `Retry`, `HeartbeatInterval` and `OnClose`; all three matter, because a proxy
because a proxy will silently drop an idle event stream. will silently drop an idle event stream.
- **gzip/deflate, decided honestly.** Iteration 36 landed the bitwise operators, - **gzip/deflate, decided honestly.** Iteration 36 landed the bitwise operators,
so a pure-`.wo` DEFLATE is now *expressible* — the question is whether it so a pure-`.wo` DEFLATE is now expressible — but the brainstorm chose C
should be. A C builtin is faster and smaller to write; a `.wo` implementation builtins for the hot path (decision 2). Negotiate, never assume: compress only
keeps the runtime doctrine intact and proves the language can do real when the client accepts the coding, only above a size threshold, and never for
bit-level work. Fork 2 decides, and the answer should turn on whether anything content that is already compressed.
else will ever want zlib.
- **Negotiate, never assume.** Compress only when the client said it accepts the ## Decisions locked (brainstorm 2026-09-06)
coding, only above a size threshold, and never for content that is already
compressed — a gzipped JPEG is bigger than the JPEG. 1. **An incoherent heartbeat/idle pair is refused at construction.** A heartbeat
interval at or above the connection's `idle_ms` evicts the very client the
heartbeat exists to keep — so the SSE endpoint refuses that pair when it is
built, with a message naming both values. A misconfiguration is a startup
error, never a flaky-network mystery — the same refuse-the-unconstructible
discipline story 6 used for streaming plus header-mutating middleware.
2. **The codec is two C builtins: `deflate` (raw) and `crc32`; gzip framing is
assembled in `.wo`.** This is the iteration's language-track work, called out
like iteration 2's phase A. The brainstorm chose performance over the
pure-`.wo` option: LZ77 match-finding and a per-byte CRC table are inner-loop
heavy and belong in C. They live in the crypto-family builtin table beside
`sha256`, hand-rolled with **no zlib dependency** (the `sha256` precedent).
`deflate` returns the raw stream; `crc32` returns the 32-bit checksum; the
gzip container (10-byte header, deflate body, little-endian CRC32 and ISIZE
trailer) is built in `.wo` with the iteration-36 bit operators. Decompression
(`inflate`) is not built — request-body decompression is out of scope.
3. **The ETag is computed over the uncompressed bytes, before compression, and
paired with `Vary: Accept-Encoding`.** `etag_for`/`with_etag` run on the
handler's uncompressed body as today; the middleware compresses afterward. The
ETag identifies the resource and is stable across encodings; `Vary` keys
caches on the encoding so a gzipped body never reaches an identity-only client;
a 304 carries no body, so the encoding is moot on a conditional hit. The
per-representation/weak-ETag alternative is more RFC-strict but makes the
middleware rewrite the handler's tag and is easy to get wrong — not chosen.
4. **`Last-Event-ID` resumption is not supported this iteration, and says so.**
Real resumption needs a per-subscriber replay buffer, which no workload asks
for yet. The client's `Last-Event-ID` is not silently ignored — that would let
a client believe it resumed when it did not; the endpoint states plainly that
it does not resume.
5. **`Vary` accumulation uses iteration 5's comma-join, not iteration 2.** This
corrects the draft: story 5 decided `Vary` accumulates by comma-joining in the
existing header map, so the compression middleware composes with other `Vary`
contributors without iteration 2's repeated-header work.
## Phases ## Phases
### Phase A — SSE framing and lifecycle ### Phase A — SSE framing and lifecycle
- The event framing (`data:`, `event:`, `id:`, `retry:`), the double-newline - The event framing (`data:`, `event:`, `id:`, `retry:`), the double-newline
terminator, and the `text/event-stream` content type with caching disabled. terminator, and the `text/event-stream` content type with caching disabled,
- Heartbeats, because an idle stream through a proxy dies quietly. Interacts emitted through iteration 6's streaming writer.
directly with iteration 35's `idle_ms` — a heartbeat interval longer than the - Heartbeats, with the construction-time refusal from decision 1 guarding the
idle deadline evicts the client the heartbeat exists to keep. interaction with iteration 35's `idle_ms`.
- Disconnect detection and cleanup: a write to a gone client must free the - Disconnect detection and cleanup: a write to a gone client must free the fiber,
fiber, the actor subscription and the fd, on every path. the actor subscription and the fd, on every path.
- Verify: a client receives ordered events; a heartbeat keeps an otherwise idle - Verify: a client receives ordered events; a heartbeat keeps an otherwise idle
stream alive past `idle_ms`; a disconnect releases everything (fd count flat). stream alive past `idle_ms`; a disconnect releases everything (fd count flat).
@ -49,42 +84,41 @@ readiness: refine
- A live feed in a sample fed by an actor, so the fan-out path is real rather - A live feed in a sample fed by an actor, so the fan-out path is real rather
than a loop in one handler. than a loop in one handler.
- `Last-Event-ID` resumption, or an explicit statement that it is not supported - `Last-Event-ID` handling per decision 4: not supported, stated explicitly.
— silently ignoring it means clients think they resumed when they did not.
- Verify: N concurrent subscribers all receive an event published once; fds and - Verify: N concurrent subscribers all receive an event published once; fds and
memory flat across a churn of connect/disconnect. memory flat across a churn of connect/disconnect.
### Phase C — the compression codec ### Phase C — the compression codec (language-track work)
- Implement or bind the codec per fork 2, with the framing gzip requires - Add the `deflate` and `crc32` builtins (decision 2), hand-rolled in C with no
(header, deflate stream, CRC32 and length trailer). Note CRC32 does not exist external dependency, documented in the builtin-surface contract and error
yet — the crypto row lists it as waiting for a consumer, and this is that catalog in the same change. CRC32 is the consumer the crypto row has listed as
consumer. waiting.
- Correctness against a reference decompressor is the acceptance bar, on - Assemble the gzip container in `.wo`: header, deflate stream, CRC32 and ISIZE
awkward input: empty, highly repetitive, incompressible, and larger than any trailer.
internal buffer. - Correctness against a reference decompressor is the acceptance bar, on awkward
- Verify: `gzip -d` reproduces the input byte-exactly for every case. input: empty, highly repetitive, incompressible, and larger than any internal
buffer.
- Verify: `oop-e2e` green for the builtins; `gzip -d` reproduces the input
byte-exactly for every case.
### Phase D — the compression middleware ### Phase D — the compression middleware
- `Accept-Encoding` negotiation reusing the q-value ranking iteration - `Accept-Encoding` negotiation reusing iteration 5's q-value ranking, a
[5](05-routing-response-ergonomics.md) added, a minimum-size threshold, and a minimum-size threshold, and a content-type skip list.
content-type skip list. - `Vary: Accept-Encoding` on anything compressed, accumulated by comma-join
- `Vary: Accept-Encoding` on anything compressed, or every cache in front will (decision 5) so it coexists with other `Vary` contributions.
serve gzipped bytes to a client that cannot read them. This needs iteration - Composition with chunked streaming: compress then chunk; the ETag rule from
2's repeated-header work to accumulate correctly with other `Vary` decision 3.
contributions. - Verify: a compressed response decompresses to the original; `Vary` is present
- Composition with chunked streaming: compress then chunk, and the ETag question and coexists; no double-compression; an already-compressed content type is
— an ETag computed over compressed bytes is a different entity than the same skipped.
resource uncompressed.
- Verify: a compressed response decompresses to the original; `Vary` is present;
no double-compression; an already-compressed content type is skipped.
### Phase E — the gate and the ledger ### Phase E — the gate and the ledger
- Both serving gates plus an ASan leg for the codec, which is the part most - Both serving gates plus an ASan leg for the codec, the part most likely to leak
likely to leak or over-read. or over-read.
- Retire the ledger's SSE and compression rows. - Retire the ledger's SSE and compression rows; record the CRC32 consumer.
- Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean. - Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean.
## Acceptance Criteria ## Acceptance Criteria
@ -93,53 +127,49 @@ readiness: refine
published, **then** the client receives them in order with correct framing. published, **then** the client receives them in order with correct framing.
- **Given** an idle SSE stream and a heartbeat interval shorter than `idle_ms`, - **Given** an idle SSE stream and a heartbeat interval shorter than `idle_ms`,
**when** it idles past the deadline, **then** it stays open. **when** it idles past the deadline, **then** it stays open.
- **Given** a heartbeat interval *longer* than `idle_ms`, **when** the stream - **Given** a heartbeat interval at or above `idle_ms`, **when** the endpoint is
idles, **then** the misconfiguration is evident rather than mysterious — the constructed, **then** it is refused with a message naming both values.
interaction is documented and, ideally, refused at construction. - **Given** a client disconnecting mid-stream, **when** the next publish occurs,
- **Given** a client disconnecting mid-stream, **when** the next publish **then** the write failure frees the fiber, the subscription and the fd; fd
occurs, **then** the write failure frees the fiber, the subscription and the count returns to baseline.
fd; fd count returns to baseline.
- **Given** N concurrent subscribers, **when** one event is published, **then** - **Given** N concurrent subscribers, **when** one event is published, **then**
all N receive it and memory does not grow per event. all N receive it and memory does not grow per event.
- **Given** any input including empty, repetitive and incompressible, **when** - **Given** any input including empty, repetitive and incompressible, **when** it
it is compressed, **then** a reference `gzip -d` reproduces it byte-exactly. is compressed, **then** a reference `gzip -d` reproduces it byte-exactly.
- **Given** a client that did not send `Accept-Encoding`, **when** it requests a - **Given** a client that did not send `Accept-Encoding`, **when** it requests a
compressible resource, **then** the response is uncompressed. compressible resource, **then** the response is uncompressed.
- **Given** a compressed response, **when** it leaves, **then** `Vary: - **Given** a compressed response, **when** it leaves, **then** `Vary:
Accept-Encoding` is set and coexists with any other `Vary` contribution. Accept-Encoding` is set and coexists with any other `Vary` contribution.
- **Given** an already-compressed content type, **when** it is served, **then** - **Given** an already-compressed content type, **when** it is served, **then**
it is not compressed again. it is not compressed again.
- **Given** a conditional request for a compressible resource, **when** its
`If-None-Match` matches the uncompressed-body ETag, **then** a 304 is returned
regardless of the client's `Accept-Encoding`.
## Out Of Scope ## Out Of Scope
- **Brotli and zstd.** One codec, proven, before a second. gzip is what every - **Brotli and zstd.** One codec, proven, before a second. gzip is what every
client accepts. client accepts.
- **Request-body decompression.** A compressed *upload* is a separate surface - **Request-body decompression (`inflate`).** A compressed *upload* is a separate
with its own decompression-bomb risk, and no workload asks yet. surface with its own decompression-bomb risk, and no workload asks yet — so the
- **WebSockets as an SSE alternative.** Already shipped and a different tool; codec builtins are compress-only.
SSE is the one-way, proxy-friendly, reconnect-by-default option. - **`Last-Event-ID` resumption** — decision 4; needs a replay buffer no workload
wants yet.
- **WebSockets as an SSE alternative.** Already shipped and a different tool; SSE
is the one-way, proxy-friendly, reconnect-by-default option.
- **Compressing static files at rest.** Fiber's static has `Compress`; - **Compressing static files at rest.** Fiber's static has `Compress`;
precompressed-file serving belongs with iteration precompressed-file serving belongs with iteration
[8](08-static-and-lifecycle.md). [8](08-static-and-lifecycle.md).
- **A general-purpose zlib library surface.** Whatever lands is what the - **A general-purpose zlib library surface.** The builtins are what the middleware
middleware needs. If a second consumer appears, it can argue for a library. needs (`deflate`, `crc32`); if a second consumer appears it can argue for
`inflate` and a library.
## Info ## Info
Forks the spec must settle: This is the second porch iteration with a language dependency (after iteration
2): two builtins, `deflate` and `crc32`, chosen over the pure-`.wo` path for
1. **Heartbeat versus idle deadline.** These two mechanisms can silently fight, hot-path performance. Everything else is pure `.wo` on iteration 6's streaming
and the failure looks like a flaky network. Decide whether the framework writer and iteration 5's negotiation. The codec is pure compute — no actors, no
refuses an incoherent pair at construction — leaning yes, because a per-key pool — so, like the rest of the track since iteration 1, it is not
configuration that cannot work should not be constructible. exposed to the lang-41 hang. The risk concentrates in the C codec (ASan leg in
2. **`.wo` DEFLATE or a C builtin?** The honest tiebreaker is whether anything phase E) and in the SSE lifecycle's fd/subscription cleanup.
else ever wants zlib. If compression is the only consumer forever, a `.wo`
implementation keeps the runtime small and is a genuine demonstration that
iteration 36's bit operators earned their place. If a second consumer is
plausible (precompressed assets, a WAL codec, an archive format), the builtin
wins. CRC32 comes along either way.
3. **ETag over compressed or uncompressed bytes?** `etag_for` exists and is used
with `with_etag` for 304s. Compressing after the ETag is computed keeps the
entity identity stable across encodings, which is almost certainly right —
but it must be decided, because getting it wrong serves the wrong body for a
conditional request.