docs(porch-sse): brainstorm story 7 (SSE + compression) to ready
- five decisions: refuse incoherent heartbeat/idle_ms pair at construction; codec = two C builtins deflate+crc32 (perf over pure-.wo; hand-rolled, no zlib dep; gzip framing in .wo); ETag over uncompressed bytes + Vary; Last-Event-ID explicitly unsupported (not silently ignored); Vary via comma-join - language enhancement: YES, two builtins -- the track's SECOND language dependency after iteration 2's random_bytes. CRC32 finally gets its consumer; inflate deliberately not built (request-body decompression OOS) - corrected stale dependency: Vary uses iteration 5's comma-join, so story 7 depends on 6 + 5, NOT 2; codec is pure compute, not lang-41-exposed - confirmed CRC32 absent + iteration 36 bit operators landed (pure-.wo was viable, traded for hot-path speed) - validated against .dev/reference/fiber. Board synced (cherry picked from commit 07f53574dd90f502235b79d4920eab3d684c8b77)
This commit is contained in:
parent
641703903c
commit
4d3e4261e1
2 changed files with 114 additions and 84 deletions
|
|
@ -1195,9 +1195,9 @@ the language arc as v1 history.
|
|||
### ▸ porch — the web framework track
|
||||
|
||||
New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md).
|
||||
Supersedes language iteration 39, now a pointer. **Stories 2–6 are `ready`
|
||||
Supersedes language iteration 39, now a pointer. **Stories 2–7 are `ready`
|
||||
(brainstormed 2026-09-06, forks locked, validated against
|
||||
`.dev/reference/fiber`); 7–8 remain `refine`.** Ordered by dependency; the
|
||||
`.dev/reference/fiber`); 8 remains `refine`.** Ordered by dependency; the
|
||||
first slice is deliberately the cheapest so the store pattern and gate shape are
|
||||
proven before the runtime and `Resp` are touched.
|
||||
|
||||
|
|
@ -1209,7 +1209,7 @@ proven before the runtime and `Resp` are touched.
|
|||
| 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 |
|
||||
| 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) |
|
||||
| 6 | [Streaming core](porch/06-streaming-core.md) | ✅ **`ready` 2026-09-06** — riskiest/highest-leverage; **re-scoped to outbound only**. Three decisions: separate `StreamHandler`/`BodyProducer` parallel path (the `Resp` path untouched → existing responses byte-identical); streaming routes **opt out** of the after-chain, framework **refuses at registration** to combine with header-mutating middleware (loud, never silent), handlers stamp headers via a `security_headers()` helper; chunked **REQUEST** bodies **split into their own future iteration** (parse.wo refusal stays). No language enhancement; rides the fiber loop, not the actor pool (not lang-41-exposed) |
|
||||
| 7 | [SSE + compression](porch/07-sse-and-compression.md) | ⬜ after 6. SSE fits the actor/fiber model unusually well; compression carries a real fork — pure-`.wo` DEFLATE (now expressible after iteration 36's bit operators) vs a C builtin. CRC32 finally gets its consumer |
|
||||
| 7 | [SSE + compression](porch/07-sse-and-compression.md) | ✅ **`ready` 2026-09-06** — after 6 (+ 5 for q-ranking/comma-join Vary; NOT 2). Five decisions: refuse an incoherent heartbeat/`idle_ms` pair at construction; **codec = two C builtins `deflate`+`crc32`** (perf over pure-`.wo`; hand-rolled, no zlib dep; gzip framing in `.wo`) — the track's **second language dependency** after iteration 2; ETag over uncompressed bytes + `Vary`; `Last-Event-ID` explicitly unsupported (not silently ignored); Vary via comma-join. Codec is pure compute — not lang-41-exposed |
|
||||
| 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip |
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -2,46 +2,81 @@
|
|||
track: porch
|
||||
iteration: "7"
|
||||
status: pending
|
||||
readiness: refine
|
||||
readiness: ready
|
||||
---
|
||||
|
||||
# porch 7 — server-sent events and compression
|
||||
|
||||
> Part of [Story — `porch`, the writeonce web framework](00-story.md).
|
||||
> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3.
|
||||
> Both halves need [iteration 6](06-streaming-core.md); neither is possible
|
||||
> before it.
|
||||
> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3,
|
||||
> re-checked 2026-09-06 against `.dev/reference/fiber` (v3, `3ca9a9d`).
|
||||
> Both halves need [iteration 6](06-streaming-core.md); the compression half also
|
||||
> needs [iteration 5](05-routing-response-ergonomics.md)'s q-value ranking and
|
||||
> comma-join `Vary`, and **two new language-track builtins** (decision 2).
|
||||
> Not dependent on iteration 2, and not exposed to the lang-41 hang.
|
||||
|
||||
## Goals
|
||||
|
||||
- **SSE, which fits this runtime unusually well.** A room actor already has the
|
||||
fan-out shape a live feed needs, and a parked fiber per subscriber costs
|
||||
almost nothing on the shard model — so the awkward part of SSE in most
|
||||
frameworks (holding thousands of idle connections) is the part writeonce
|
||||
already solved with iteration 35's idle deadlines and fiber-per-connection.
|
||||
Fiber ships `Retry`, `HeartbeatInterval` and `OnClose`; all three matter,
|
||||
because a proxy will silently drop an idle event stream.
|
||||
fan-out shape a live feed needs, and a parked fiber per subscriber costs almost
|
||||
nothing on the shard model — so the awkward part of SSE in most frameworks
|
||||
(holding thousands of idle connections) is the part writeonce already solved
|
||||
with iteration 35's idle deadlines and fiber-per-connection. Fiber ships
|
||||
`Retry`, `HeartbeatInterval` and `OnClose`; all three matter, because a proxy
|
||||
will silently drop an idle event stream.
|
||||
- **gzip/deflate, decided honestly.** Iteration 36 landed the bitwise operators,
|
||||
so a pure-`.wo` DEFLATE is now *expressible* — the question is whether it
|
||||
should be. A C builtin is faster and smaller to write; a `.wo` implementation
|
||||
keeps the runtime doctrine intact and proves the language can do real
|
||||
bit-level work. Fork 2 decides, and the answer should turn on whether anything
|
||||
else will ever want zlib.
|
||||
- **Negotiate, never assume.** Compress only when the client said it accepts the
|
||||
coding, only above a size threshold, and never for content that is already
|
||||
compressed — a gzipped JPEG is bigger than the JPEG.
|
||||
so a pure-`.wo` DEFLATE is now expressible — but the brainstorm chose C
|
||||
builtins for the hot path (decision 2). Negotiate, never assume: compress only
|
||||
when the client accepts the coding, only above a size threshold, and never for
|
||||
content that is already compressed.
|
||||
|
||||
## Decisions locked (brainstorm 2026-09-06)
|
||||
|
||||
1. **An incoherent heartbeat/idle pair is refused at construction.** A heartbeat
|
||||
interval at or above the connection's `idle_ms` evicts the very client the
|
||||
heartbeat exists to keep — so the SSE endpoint refuses that pair when it is
|
||||
built, with a message naming both values. A misconfiguration is a startup
|
||||
error, never a flaky-network mystery — the same refuse-the-unconstructible
|
||||
discipline story 6 used for streaming plus header-mutating middleware.
|
||||
2. **The codec is two C builtins: `deflate` (raw) and `crc32`; gzip framing is
|
||||
assembled in `.wo`.** This is the iteration's language-track work, called out
|
||||
like iteration 2's phase A. The brainstorm chose performance over the
|
||||
pure-`.wo` option: LZ77 match-finding and a per-byte CRC table are inner-loop
|
||||
heavy and belong in C. They live in the crypto-family builtin table beside
|
||||
`sha256`, hand-rolled with **no zlib dependency** (the `sha256` precedent).
|
||||
`deflate` returns the raw stream; `crc32` returns the 32-bit checksum; the
|
||||
gzip container (10-byte header, deflate body, little-endian CRC32 and ISIZE
|
||||
trailer) is built in `.wo` with the iteration-36 bit operators. Decompression
|
||||
(`inflate`) is not built — request-body decompression is out of scope.
|
||||
3. **The ETag is computed over the uncompressed bytes, before compression, and
|
||||
paired with `Vary: Accept-Encoding`.** `etag_for`/`with_etag` run on the
|
||||
handler's uncompressed body as today; the middleware compresses afterward. The
|
||||
ETag identifies the resource and is stable across encodings; `Vary` keys
|
||||
caches on the encoding so a gzipped body never reaches an identity-only client;
|
||||
a 304 carries no body, so the encoding is moot on a conditional hit. The
|
||||
per-representation/weak-ETag alternative is more RFC-strict but makes the
|
||||
middleware rewrite the handler's tag and is easy to get wrong — not chosen.
|
||||
4. **`Last-Event-ID` resumption is not supported this iteration, and says so.**
|
||||
Real resumption needs a per-subscriber replay buffer, which no workload asks
|
||||
for yet. The client's `Last-Event-ID` is not silently ignored — that would let
|
||||
a client believe it resumed when it did not; the endpoint states plainly that
|
||||
it does not resume.
|
||||
5. **`Vary` accumulation uses iteration 5's comma-join, not iteration 2.** This
|
||||
corrects the draft: story 5 decided `Vary` accumulates by comma-joining in the
|
||||
existing header map, so the compression middleware composes with other `Vary`
|
||||
contributors without iteration 2's repeated-header work.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase A — SSE framing and lifecycle
|
||||
|
||||
- The event framing (`data:`, `event:`, `id:`, `retry:`), the double-newline
|
||||
terminator, and the `text/event-stream` content type with caching disabled.
|
||||
- Heartbeats, because an idle stream through a proxy dies quietly. Interacts
|
||||
directly with iteration 35's `idle_ms` — a heartbeat interval longer than the
|
||||
idle deadline evicts the client the heartbeat exists to keep.
|
||||
- Disconnect detection and cleanup: a write to a gone client must free the
|
||||
fiber, the actor subscription and the fd, on every path.
|
||||
terminator, and the `text/event-stream` content type with caching disabled,
|
||||
emitted through iteration 6's streaming writer.
|
||||
- Heartbeats, with the construction-time refusal from decision 1 guarding the
|
||||
interaction with iteration 35's `idle_ms`.
|
||||
- Disconnect detection and cleanup: a write to a gone client must free the fiber,
|
||||
the actor subscription and the fd, on every path.
|
||||
- Verify: a client receives ordered events; a heartbeat keeps an otherwise idle
|
||||
stream alive past `idle_ms`; a disconnect releases everything (fd count flat).
|
||||
|
||||
|
|
@ -49,42 +84,41 @@ readiness: refine
|
|||
|
||||
- A live feed in a sample fed by an actor, so the fan-out path is real rather
|
||||
than a loop in one handler.
|
||||
- `Last-Event-ID` resumption, or an explicit statement that it is not supported
|
||||
— silently ignoring it means clients think they resumed when they did not.
|
||||
- `Last-Event-ID` handling per decision 4: not supported, stated explicitly.
|
||||
- Verify: N concurrent subscribers all receive an event published once; fds and
|
||||
memory flat across a churn of connect/disconnect.
|
||||
|
||||
### Phase C — the compression codec
|
||||
### Phase C — the compression codec (language-track work)
|
||||
|
||||
- Implement or bind the codec per fork 2, with the framing gzip requires
|
||||
(header, deflate stream, CRC32 and length trailer). Note CRC32 does not exist
|
||||
yet — the crypto row lists it as waiting for a consumer, and this is that
|
||||
consumer.
|
||||
- Correctness against a reference decompressor is the acceptance bar, on
|
||||
awkward input: empty, highly repetitive, incompressible, and larger than any
|
||||
internal buffer.
|
||||
- Verify: `gzip -d` reproduces the input byte-exactly for every case.
|
||||
- Add the `deflate` and `crc32` builtins (decision 2), hand-rolled in C with no
|
||||
external dependency, documented in the builtin-surface contract and error
|
||||
catalog in the same change. CRC32 is the consumer the crypto row has listed as
|
||||
waiting.
|
||||
- Assemble the gzip container in `.wo`: header, deflate stream, CRC32 and ISIZE
|
||||
trailer.
|
||||
- Correctness against a reference decompressor is the acceptance bar, on awkward
|
||||
input: empty, highly repetitive, incompressible, and larger than any internal
|
||||
buffer.
|
||||
- Verify: `oop-e2e` green for the builtins; `gzip -d` reproduces the input
|
||||
byte-exactly for every case.
|
||||
|
||||
### Phase D — the compression middleware
|
||||
|
||||
- `Accept-Encoding` negotiation reusing the q-value ranking iteration
|
||||
[5](05-routing-response-ergonomics.md) added, a minimum-size threshold, and a
|
||||
content-type skip list.
|
||||
- `Vary: Accept-Encoding` on anything compressed, or every cache in front will
|
||||
serve gzipped bytes to a client that cannot read them. This needs iteration
|
||||
2's repeated-header work to accumulate correctly with other `Vary`
|
||||
contributions.
|
||||
- Composition with chunked streaming: compress then chunk, and the ETag question
|
||||
— an ETag computed over compressed bytes is a different entity than the same
|
||||
resource uncompressed.
|
||||
- Verify: a compressed response decompresses to the original; `Vary` is present;
|
||||
no double-compression; an already-compressed content type is skipped.
|
||||
- `Accept-Encoding` negotiation reusing iteration 5's q-value ranking, a
|
||||
minimum-size threshold, and a content-type skip list.
|
||||
- `Vary: Accept-Encoding` on anything compressed, accumulated by comma-join
|
||||
(decision 5) so it coexists with other `Vary` contributions.
|
||||
- Composition with chunked streaming: compress then chunk; the ETag rule from
|
||||
decision 3.
|
||||
- Verify: a compressed response decompresses to the original; `Vary` is present
|
||||
and coexists; no double-compression; an already-compressed content type is
|
||||
skipped.
|
||||
|
||||
### Phase E — the gate and the ledger
|
||||
|
||||
- Both serving gates plus an ASan leg for the codec, which is the part most
|
||||
likely to leak or over-read.
|
||||
- Retire the ledger's SSE and compression rows.
|
||||
- Both serving gates plus an ASan leg for the codec, the part most likely to leak
|
||||
or over-read.
|
||||
- Retire the ledger's SSE and compression rows; record the CRC32 consumer.
|
||||
- Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
|
@ -93,53 +127,49 @@ readiness: refine
|
|||
published, **then** the client receives them in order with correct framing.
|
||||
- **Given** an idle SSE stream and a heartbeat interval shorter than `idle_ms`,
|
||||
**when** it idles past the deadline, **then** it stays open.
|
||||
- **Given** a heartbeat interval *longer* than `idle_ms`, **when** the stream
|
||||
idles, **then** the misconfiguration is evident rather than mysterious — the
|
||||
interaction is documented and, ideally, refused at construction.
|
||||
- **Given** a client disconnecting mid-stream, **when** the next publish
|
||||
occurs, **then** the write failure frees the fiber, the subscription and the
|
||||
fd; fd count returns to baseline.
|
||||
- **Given** a heartbeat interval at or above `idle_ms`, **when** the endpoint is
|
||||
constructed, **then** it is refused with a message naming both values.
|
||||
- **Given** a client disconnecting mid-stream, **when** the next publish occurs,
|
||||
**then** the write failure frees the fiber, the subscription and the fd; fd
|
||||
count returns to baseline.
|
||||
- **Given** N concurrent subscribers, **when** one event is published, **then**
|
||||
all N receive it and memory does not grow per event.
|
||||
- **Given** any input including empty, repetitive and incompressible, **when**
|
||||
it is compressed, **then** a reference `gzip -d` reproduces it byte-exactly.
|
||||
- **Given** any input including empty, repetitive and incompressible, **when** it
|
||||
is compressed, **then** a reference `gzip -d` reproduces it byte-exactly.
|
||||
- **Given** a client that did not send `Accept-Encoding`, **when** it requests a
|
||||
compressible resource, **then** the response is uncompressed.
|
||||
- **Given** a compressed response, **when** it leaves, **then** `Vary:
|
||||
Accept-Encoding` is set and coexists with any other `Vary` contribution.
|
||||
- **Given** an already-compressed content type, **when** it is served, **then**
|
||||
it is not compressed again.
|
||||
- **Given** a conditional request for a compressible resource, **when** its
|
||||
`If-None-Match` matches the uncompressed-body ETag, **then** a 304 is returned
|
||||
regardless of the client's `Accept-Encoding`.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- **Brotli and zstd.** One codec, proven, before a second. gzip is what every
|
||||
client accepts.
|
||||
- **Request-body decompression.** A compressed *upload* is a separate surface
|
||||
with its own decompression-bomb risk, and no workload asks yet.
|
||||
- **WebSockets as an SSE alternative.** Already shipped and a different tool;
|
||||
SSE is the one-way, proxy-friendly, reconnect-by-default option.
|
||||
- **Request-body decompression (`inflate`).** A compressed *upload* is a separate
|
||||
surface with its own decompression-bomb risk, and no workload asks yet — so the
|
||||
codec builtins are compress-only.
|
||||
- **`Last-Event-ID` resumption** — decision 4; needs a replay buffer no workload
|
||||
wants yet.
|
||||
- **WebSockets as an SSE alternative.** Already shipped and a different tool; SSE
|
||||
is the one-way, proxy-friendly, reconnect-by-default option.
|
||||
- **Compressing static files at rest.** Fiber's static has `Compress`;
|
||||
precompressed-file serving belongs with iteration
|
||||
[8](08-static-and-lifecycle.md).
|
||||
- **A general-purpose zlib library surface.** Whatever lands is what the
|
||||
middleware needs. If a second consumer appears, it can argue for a library.
|
||||
- **A general-purpose zlib library surface.** The builtins are what the middleware
|
||||
needs (`deflate`, `crc32`); if a second consumer appears it can argue for
|
||||
`inflate` and a library.
|
||||
|
||||
## Info
|
||||
|
||||
Forks the spec must settle:
|
||||
|
||||
1. **Heartbeat versus idle deadline.** These two mechanisms can silently fight,
|
||||
and the failure looks like a flaky network. Decide whether the framework
|
||||
refuses an incoherent pair at construction — leaning yes, because a
|
||||
configuration that cannot work should not be constructible.
|
||||
2. **`.wo` DEFLATE or a C builtin?** The honest tiebreaker is whether anything
|
||||
else ever wants zlib. If compression is the only consumer forever, a `.wo`
|
||||
implementation keeps the runtime small and is a genuine demonstration that
|
||||
iteration 36's bit operators earned their place. If a second consumer is
|
||||
plausible (precompressed assets, a WAL codec, an archive format), the builtin
|
||||
wins. CRC32 comes along either way.
|
||||
3. **ETag over compressed or uncompressed bytes?** `etag_for` exists and is used
|
||||
with `with_etag` for 304s. Compressing after the ETag is computed keeps the
|
||||
entity identity stable across encodings, which is almost certainly right —
|
||||
but it must be decided, because getting it wrong serves the wrong body for a
|
||||
conditional request.
|
||||
This is the second porch iteration with a language dependency (after iteration
|
||||
2): two builtins, `deflate` and `crc32`, chosen over the pure-`.wo` path for
|
||||
hot-path performance. Everything else is pure `.wo` on iteration 6's streaming
|
||||
writer and iteration 5's negotiation. The codec is pure compute — no actors, no
|
||||
per-key pool — so, like the rest of the track since iteration 1, it is not
|
||||
exposed to the lang-41 hang. The risk concentrates in the C codec (ASan leg in
|
||||
phase E) and in the SSE lifecycle's fd/subscription cleanup.
|
||||
|
|
|
|||
Loading…
Reference in a new issue