From 641703903c680a214b45f72129a4885f9f4c153e Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Sun, 6 Sep 2026 17:31:06 +0200 Subject: [PATCH] docs(porch-streaming): brainstorm story 6 (streaming core) to ready - re-scoped to OUTBOUND streaming only - three decisions: separate StreamHandler/BodyProducer parallel path (Resp path untouched -> existing responses byte-identical); streaming routes opt out of the after-chain, framework refuses at registration to combine with header-mutating middleware (loud, never silent), security_headers() helper lets handlers stamp them; chunked REQUEST bodies split into their own future iteration (parse.wo refusal stays, smuggling cases enumerated for later) - no language enhancement (net.write framing, fs.read_at/actor source, interfaces for producer); rides the fiber loop not the actor pool, so not lang-41-exposed - fixed title inconsistency: "three iterations wait on" -> "two" (7 and 8) - validated against .dev/reference/fiber + the app.wo/serve.wo pipeline. Board synced (cherry picked from commit 15205408e03c3c02a38e00e5d2017a8a11f62f28) --- docs/stories/00-status.md | 8 +- docs/stories/porch/06-streaming-core.md | 182 ++++++++++++++---------- 2 files changed, 110 insertions(+), 80 deletions(-) diff --git a/docs/stories/00-status.md b/docs/stories/00-status.md index 3bae234..f494061 100644 --- a/docs/stories/00-status.md +++ b/docs/stories/00-status.md @@ -1195,9 +1195,9 @@ the language arc as v1 history. ### ▸ porch — the web framework track New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md). -Supersedes language iteration 39, now a pointer. **Stories 2, 3, 4 and 5 are -`ready` (brainstormed 2026-09-06, forks locked, validated against -`.dev/reference/fiber`); 6–8 remain `refine`.** Ordered by dependency; the +Supersedes language iteration 39, now a pointer. **Stories 2–6 are `ready` +(brainstormed 2026-09-06, forks locked, validated against +`.dev/reference/fiber`); 7–8 remain `refine`.** Ordered by dependency; the first slice is deliberately the cheapest so the store pattern and gate shape are proven before the runtime and `Resp` are touched. @@ -1208,7 +1208,7 @@ proven before the runtime and `Resp` are touched. | 3 | [Sessions](porch/03-sessions.md) | ✅ **`ready` 2026-09-06** — after 2. Six decisions locked: row is a pure auth primitive (id/principal/created_at/last_seen, no payload bag); **wall-clock `time.now`, not monotonic `time.ticks`** (sessions survive restart); rotation = login always mints a fresh id (no anon-session model); throttled `last_seen` touch at `idle/20` (not a WAL write per request); `Session` writes `req.principal`; config refuses absolute < idle. Pure `.wo` on iteration 2 + the `@table` engine — no new runtime work | | 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 | | 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) | -| 6 | [Streaming core](porch/06-streaming-core.md) | ⬜ the riskiest and highest-leverage slice: incremental writes + chunked framing + an explicit commit point. `serialize()` always emits `Content-Length` today. Chunked REQUEST bodies are deliberately refused (request smuggling) and that refusal must survive | +| 6 | [Streaming core](porch/06-streaming-core.md) | ✅ **`ready` 2026-09-06** — riskiest/highest-leverage; **re-scoped to outbound only**. Three decisions: separate `StreamHandler`/`BodyProducer` parallel path (the `Resp` path untouched → existing responses byte-identical); streaming routes **opt out** of the after-chain, framework **refuses at registration** to combine with header-mutating middleware (loud, never silent), handlers stamp headers via a `security_headers()` helper; chunked **REQUEST** bodies **split into their own future iteration** (parse.wo refusal stays). No language enhancement; rides the fiber loop, not the actor pool (not lang-41-exposed) | | 7 | [SSE + compression](porch/07-sse-and-compression.md) | ⬜ after 6. SSE fits the actor/fiber model unusually well; compression carries a real fork — pure-`.wo` DEFLATE (now expressible after iteration 36's bit operators) vs a C builtin. CRC32 finally gets its consumer | | 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip | diff --git a/docs/stories/porch/06-streaming-core.md b/docs/stories/porch/06-streaming-core.md index 417ed6e..5d3c7e7 100644 --- a/docs/stories/porch/06-streaming-core.md +++ b/docs/stories/porch/06-streaming-core.md @@ -2,19 +2,25 @@ track: porch iteration: "6" status: pending -readiness: refine +readiness: ready --- -# porch 6 — streaming core: the seam three iterations wait on +# porch 6 — streaming core: the seam two iterations wait on > Part of [Story — `porch`, the writeonce web framework](00-story.md). -> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3. +> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3, +> re-checked 2026-09-06 against `.dev/reference/fiber` (v3, `3ca9a9d`) and the +> existing `internal/serve.wo`/`app.wo` dispatch pipeline. > > The largest and riskiest slice in this track, and the one with the most > downstream value: iterations [7](07-sse-and-compression.md) and > [8](08-static-and-lifecycle.md) are both blocked on it, and porch's README has > carried "lazy body streaming · streaming responses · explicit commit point" > as parked since framework v1. +> +> Re-scoped by the brainstorm to **outbound streaming only** — chunked *request* +> bodies split into their own future iteration (decision 3), so this slice ships +> and gates without entangling the smuggling surface. ## Goals @@ -27,69 +33,91 @@ readiness: refine - **Chunked transfer-encoding on the way out**, correctly framed and correctly terminated, because a truncated chunked response is indistinguishable from a network failure to the client and corrupts keep-alive for the connection. -- **Chunked request bodies on the way in — carefully.** - `internal/parse.wo` **deliberately refuses** them today, with a correct note - that silently treating a chunked request as body-less is request smuggling. - That refusal is good engineering. It may only be lifted by an implementation - that handles the smuggling cases explicitly, and the refusal must remain the - behaviour for anything the parser is not certain about. - **An explicit commit point.** Once the first byte is written, the status and headers are gone and no `after` middleware can change them. That is a real semantic change to the middleware contract and it has to be stated, not - discovered — the `after` chain currently runs on *every* response and - security headers depend on it. + discovered — the `after` chain currently runs on *every* response and security + headers depend on it. + +## Decisions locked (brainstorm 2026-09-06) + +1. **A separate `StreamHandler` interface and a parallel dispatch path — the + `Resp` path is untouched.** A streaming route registers a `StreamHandler` + (`app.stream(pattern, …)`), distinct from `Handler`. Its output is a stream + outcome carrying status, a header set, and a `BodyProducer` — an interface + (a class with a `next() -> ?Bytes` method, `nil` = done), so the no-closures + doctrine holds. Dispatch resolves a streaming route on its own branch; + `serve_conn` gains one branch: a stream outcome writes its headers then pumps + the producer as chunks, everything else serializes exactly as today. The two + paths coexist, which is what keeps every existing response byte-identical. +2. **Streaming routes opt out of the after-chain, and the framework refuses at + registration to combine one with header-mutating middleware.** The after-chain + is *not* re-plumbed onto the streaming branch. Instead, at startup the + framework raises a loud error naming the conflict if a streaming route is in + the scope of a header-mutating `after` middleware (`SecurityHeaders`, `Cors`) + — never a silent partial application, which the study calls the one + unacceptable answer. A stream handler that wants those headers stamps them + itself via a `security_headers()` helper before committing, so opting out of + the chain does not mean losing them — it means setting them explicitly. Once + `serve_conn` writes the header set, it is frozen; a later mutation attempt + hits a guard that traps or logs in development. +3. **Chunked *request*-body parsing is split into its own future iteration; the + deliberate refusal stays until then.** Inbound parsing is orthogonal to + outbound streaming (a different file, `parse.wo`), it is the single riskiest + security surface in the framework (smuggling), and this slice is already the + largest — so it ships outbound-only. `internal/parse.wo`'s refusal of chunked + request bodies remains the behaviour, correct as it stands. ## Phases ### Phase A — the writer seam -- Decide the shape (fork 1) and introduce a way for a handler to emit body - bytes progressively instead of returning a complete `Resp`. Handlers are - classes, so this is a second interface beside `Handler`, not a callback. +- Introduce the `StreamHandler` and `BodyProducer` interfaces and the + `app.stream` registration (decision 1). A streaming handler emits body bytes + progressively through the producer instead of returning a complete `Resp`. - Keep the existing whole-response path as the default and unchanged: the - overwhelming majority of responses are small and should not pay for this. -- Verify: the two paths coexist; every existing response is byte-identical. + overwhelming majority of responses are small and must not pay for this. The + `Resp` path, `serialize()`, `route_req` and the after-chain are untouched. +- Verify: the two paths coexist; every existing response is byte-identical, both + serving gates unchanged. ### Phase B — chunked responses -- Chunk framing, the terminating zero-length chunk, and the interaction with - keep-alive — a connection whose chunked response was truncated must be closed, - not reused. -- `Content-Length` and chunked are mutually exclusive; `serialize()` must pick - one and never emit both. -- `HEAD` on a streaming route: headers only, and decide what `Content-Length` - claims when the length is unknown. +- Chunk framing (hex size, CRLF, bytes, CRLF), the terminating zero-length + chunk, and `Transfer-Encoding: chunked` in the header set; `Content-Length` + and chunked are mutually exclusive and the stream path emits chunked and never + a length — mutual exclusion holds by construction because it is a separate path + from `serialize()`. +- Keep-alive interaction: a chunked response that completes (zero chunk sent) + may keep the connection; one truncated by a mid-stream producer trap must close + it, never reuse it, and leave no half-frame. +- `HEAD` on a streaming route: the header set (including `Transfer-Encoding: + chunked`) with no body and no `Content-Length`, since the length is unknown. - Verify: a chunked response reassembles byte-exactly; a mid-stream trap closes the connection rather than leaving a half-frame; `HEAD` is coherent. ### Phase C — the commit point and the middleware contract -- Define and enforce when headers are locked. An `after` middleware that tries - to mutate a committed response must fail loudly in development rather than - silently doing nothing. -- Decide what happens to `SecurityHeaders` and `Cors` — both are `after` - middleware and both must still apply to streamed responses, which means they - have to run *before* the commit for those routes. -- Verify: security headers and CORS are present on a streamed response; a - post-commit mutation attempt is reported. +- Implement the opt-out and the registration refusal (decision 2): a streaming + route under a header-mutating `after` middleware is refused at startup with a + message naming the conflict. +- Provide the `security_headers()` helper so a stream handler stamps the standard + security headers into its own header set before committing. +- Enforce the commit point: after `serve_conn` writes the header set, it is + frozen; a mutation attempt traps or logs in development rather than silently + doing nothing. +- Verify: security headers and CORS are present on a streamed response that asks + for them via the helper; a streaming route wrongly combined with + header-mutating middleware is refused at startup; a post-commit mutation + attempt is reported. -### Phase D — chunked request bodies - -- Only if phase C is clean. Parse chunked request bodies with the smuggling - cases enumerated and tested: both `Content-Length` and `Transfer-Encoding` - present, duplicated `Transfer-Encoding`, unknown transfer codings, and - oversized or malformed chunk sizes. -- Every ambiguous case stays a 400-and-close, matching the existing duplicate - `Content-Length` discipline. -- Verify: a well-formed chunked upload arrives intact; every enumerated - smuggling shape is refused. - -### Phase E — the gate and the ledger +### Phase D — the gate and the ledger - A streaming route in a sample, gated on both consumers, plus a large-body leg - proving memory does not scale with response size. + proving peak memory does not scale with response size, and a regression check + that the WebSocket 101 hijack path still works untouched. - Retire the README's parked streaming rows; record the commit-point semantics - where a handler author will find them. + and the streaming opt-out where a handler author will find them. - Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean. ## Acceptance Criteria @@ -103,54 +131,56 @@ readiness: refine - **Given** a handler that traps mid-stream, **when** the failure occurs, **then** the connection is closed rather than reused, no half-frame is left behind, and the server survives. -- **Given** a streamed response, **when** it leaves, **then** the security and - CORS headers the `after` chain contributes are still present. -- **Given** an `after` middleware attempting to change a committed response, +- **Given** a streamed response whose handler calls `security_headers()`, + **when** it leaves, **then** those headers are present. +- **Given** a streaming route registered in the scope of a header-mutating + `after` middleware, **when** the app starts, **then** it is refused with a + message naming the conflict — never a silently unheadered stream. +- **Given** an `after` or handler attempting to change a committed response, **when** it runs, **then** the attempt is reported rather than silently dropped. - **Given** a `HEAD` request to a streaming route, **when** it is answered, - **then** the headers are coherent and no body is sent. -- **Given** a request with both `Content-Length` and `Transfer-Encoding`, - **when** it is parsed, **then** it is refused with 400 and the connection is - closed — smuggling is refused, never guessed at. + **then** the headers are coherent (chunked, no `Content-Length`) and no body is + sent. +- **Given** the WebSocket 101 hijack path, **when** an upgrade is handled after + this iteration, **then** it still bypasses `serialize()` and works untouched. - **Given** every pre-existing non-streaming response, **when** both serving gates run, **then** output is byte-identical to before this iteration. ## Out Of Scope +- **Chunked *request* bodies** — split into their own future iteration + (decision 3). The `internal/parse.wo` refusal stays until then; when it is + picked up it must enumerate the smuggling cases (both `Content-Length` and + `Transfer-Encoding` present, duplicated `Transfer-Encoding`, unknown transfer + codings, oversized or malformed chunk sizes), each a 400-and-close, matching + the existing duplicate `Content-Length` discipline. - **SSE** — iteration [7](07-sse-and-compression.md), the first consumer. - **Compression** — also [7](07-sse-and-compression.md); it composes with chunking and should not be entangled with building it. -- **Byte ranges and `SendFile`** — iteration - [8](08-static-and-lifecycle.md). +- **Byte ranges and `SendFile`** — iteration [8](08-static-and-lifecycle.md). - **Request-body backpressure as a general mechanism.** Reading a body slowly to - push back on a producer wants cancellation, which porch does not have and - which language iteration 31's actor lifecycle owns. This iteration streams - *out* and parses chunked *in*; it does not add flow control. + push back on a producer wants cancellation, which porch does not have and which + language iteration 31's actor lifecycle owns. This iteration streams *out*; it + does not add flow control. A slow client simply parks the writing fiber. - **WebSockets.** Already shipped (`ws_accept`, `wsframe`) and deliberately a hijack that bypasses `serialize()` — that path must keep working untouched, - which is worth an explicit regression check. + which is why phase D checks it. - **HTTP/2.** Proxy-terminated by doctrine, and parked behind language iteration 23 regardless. ## Info -Forks the spec must settle: +No language enhancement is needed: the writer seam is `net.write` (id 54) called +repeatedly for framing, a `BodyProducer` sourcing bytes from `fs.read_at` +(id 44) or an actor `receive`, and interfaces/classes for the producer — all +present. The risk in this slice is design risk (the commit contract, framing +correctness, keep-alive on truncation), not a runtime gap, and the streaming +writes ride the existing fiber-per-connection loop, not the per-key actor pool, +so this iteration is not exposed to the lang-41 hang. -1. **What is the writer?** Candidates: a second interface whose method is - called repeatedly until it signals done; a `Resp` variant carrying a producer - object instead of a `Text` body; or a handler that receives the connection and - writes directly (which is what `ws_accept` already does via the 101 hijack - sentinel). The third is the least new machinery and the most footgun. The - first fits the no-closures doctrine best, since a producer is just another - class with fields. -2. **Does the `after` chain still run for streamed responses?** It must, or - security headers regress. But it cannot run *after* the body. So either - `after` runs at commit time for streaming routes, or streaming routes declare - they opt out and the framework refuses to combine them with header-mutating - middleware. Silent partial application is the one unacceptable answer. -3. **Is chunked request parsing in this iteration at all?** It is separable and - it is the riskiest security surface in the framework. Splitting phase D into - its own iteration is a legitimate outcome of the brainstorm — the study is - explicit that the current refusal is *correct*, so there is no pressure to - rush it. +Forks are settled above. The one the brainstorm moved most was fork 1: rather +than folding a producer into `Resp` (which would have made the after-chain apply +for free), the decision is a genuinely separate `StreamHandler` path — which is +why fork 2 had to be answered explicitly, and why streaming routes opt out of +the after-chain with a loud registration refusal rather than inheriting it.