docs(porch-streaming): brainstorm story 6 (streaming core) to ready

- re-scoped to OUTBOUND streaming only
- three decisions: separate StreamHandler/BodyProducer parallel path (Resp
  path untouched -> existing responses byte-identical); streaming routes opt
  out of the after-chain, framework refuses at registration to combine with
  header-mutating middleware (loud, never silent), security_headers() helper
  lets handlers stamp them; chunked REQUEST bodies split into their own future
  iteration (parse.wo refusal stays, smuggling cases enumerated for later)
- no language enhancement (net.write framing, fs.read_at/actor source,
  interfaces for producer); rides the fiber loop not the actor pool, so not
  lang-41-exposed
- fixed title inconsistency: "three iterations wait on" -> "two" (7 and 8)
- validated against .dev/reference/fiber + the app.wo/serve.wo pipeline. Board synced

(cherry picked from commit 15205408e03c3c02a38e00e5d2017a8a11f62f28)
This commit is contained in:
shoney.arickathil 2026-09-06 17:31:06 +02:00
parent 6b820fdd5f
commit 641703903c
2 changed files with 110 additions and 80 deletions

View file

@ -1195,9 +1195,9 @@ the language arc as v1 history.
### ▸ porch — the web framework track ### ▸ porch — the web framework track
New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md). New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md).
Supersedes language iteration 39, now a pointer. **Stories 2, 3, 4 and 5 are Supersedes language iteration 39, now a pointer. **Stories 2–6 are `ready`
`ready` (brainstormed 2026-09-06, forks locked, validated against (brainstormed 2026-09-06, forks locked, validated against
`.dev/reference/fiber`); 6–8 remain `refine`.** Ordered by dependency; the `.dev/reference/fiber`); 7–8 remain `refine`.** Ordered by dependency; the
first slice is deliberately the cheapest so the store pattern and gate shape are first slice is deliberately the cheapest so the store pattern and gate shape are
proven before the runtime and `Resp` are touched. proven before the runtime and `Resp` are touched.
@ -1208,7 +1208,7 @@ proven before the runtime and `Resp` are touched.
| 3 | [Sessions](porch/03-sessions.md) | ✅ **`ready` 2026-09-06** — after 2. Six decisions locked: row is a pure auth primitive (id/principal/created_at/last_seen, no payload bag); **wall-clock `time.now`, not monotonic `time.ticks`** (sessions survive restart); rotation = login always mints a fresh id (no anon-session model); throttled `last_seen` touch at `idle/20` (not a WAL write per request); `Session` writes `req.principal`; config refuses absolute < idle. Pure `.wo` on iteration 2 + the `@table` engine — no new runtime work | | 3 | [Sessions](porch/03-sessions.md) | ✅ **`ready` 2026-09-06** — after 2. Six decisions locked: row is a pure auth primitive (id/principal/created_at/last_seen, no payload bag); **wall-clock `time.now`, not monotonic `time.ticks`** (sessions survive restart); rotation = login always mints a fresh id (no anon-session model); throttled `last_seen` touch at `idle/20` (not a WAL write per request); `Session` writes `req.principal`; config refuses absolute < idle. Pure `.wo` on iteration 2 + the `@table` engine — no new runtime work |
| 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 | | 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 |
| 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) | | 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) |
| 6 | [Streaming core](porch/06-streaming-core.md) | ⬜ the riskiest and highest-leverage slice: incremental writes + chunked framing + an explicit commit point. `serialize()` always emits `Content-Length` today. Chunked REQUEST bodies are deliberately refused (request smuggling) and that refusal must survive | | 6 | [Streaming core](porch/06-streaming-core.md) | ✅ **`ready` 2026-09-06** — riskiest/highest-leverage; **re-scoped to outbound only**. Three decisions: separate `StreamHandler`/`BodyProducer` parallel path (the `Resp` path untouched → existing responses byte-identical); streaming routes **opt out** of the after-chain, framework **refuses at registration** to combine with header-mutating middleware (loud, never silent), handlers stamp headers via a `security_headers()` helper; chunked **REQUEST** bodies **split into their own future iteration** (parse.wo refusal stays). No language enhancement; rides the fiber loop, not the actor pool (not lang-41-exposed) |
| 7 | [SSE + compression](porch/07-sse-and-compression.md) | ⬜ after 6. SSE fits the actor/fiber model unusually well; compression carries a real fork — pure-`.wo` DEFLATE (now expressible after iteration 36's bit operators) vs a C builtin. CRC32 finally gets its consumer | | 7 | [SSE + compression](porch/07-sse-and-compression.md) | ⬜ after 6. SSE fits the actor/fiber model unusually well; compression carries a real fork — pure-`.wo` DEFLATE (now expressible after iteration 36's bit operators) vs a C builtin. CRC32 finally gets its consumer |
| 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip | | 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip |

View file

@ -2,19 +2,25 @@
track: porch track: porch
iteration: "6" iteration: "6"
status: pending status: pending
readiness: refine readiness: ready
--- ---
# porch 6 — streaming core: the seam three iterations wait on # porch 6 — streaming core: the seam two iterations wait on
> Part of [Story — `porch`, the writeonce web framework](00-story.md). > Part of [Story — `porch`, the writeonce web framework](00-story.md).
> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3. > Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3,
> re-checked 2026-09-06 against `.dev/reference/fiber` (v3, `3ca9a9d`) and the
> existing `internal/serve.wo`/`app.wo` dispatch pipeline.
> >
> The largest and riskiest slice in this track, and the one with the most > The largest and riskiest slice in this track, and the one with the most
> downstream value: iterations [7](07-sse-and-compression.md) and > downstream value: iterations [7](07-sse-and-compression.md) and
> [8](08-static-and-lifecycle.md) are both blocked on it, and porch's README has > [8](08-static-and-lifecycle.md) are both blocked on it, and porch's README has
> carried "lazy body streaming · streaming responses · explicit commit point" > carried "lazy body streaming · streaming responses · explicit commit point"
> as parked since framework v1. > as parked since framework v1.
>
> Re-scoped by the brainstorm to **outbound streaming only** — chunked *request*
> bodies split into their own future iteration (decision 3), so this slice ships
> and gates without entangling the smuggling surface.
## Goals ## Goals
@ -27,69 +33,91 @@ readiness: refine
- **Chunked transfer-encoding on the way out**, correctly framed and correctly - **Chunked transfer-encoding on the way out**, correctly framed and correctly
terminated, because a truncated chunked response is indistinguishable from a terminated, because a truncated chunked response is indistinguishable from a
network failure to the client and corrupts keep-alive for the connection. network failure to the client and corrupts keep-alive for the connection.
- **Chunked request bodies on the way in — carefully.**
`internal/parse.wo` **deliberately refuses** them today, with a correct note
that silently treating a chunked request as body-less is request smuggling.
That refusal is good engineering. It may only be lifted by an implementation
that handles the smuggling cases explicitly, and the refusal must remain the
behaviour for anything the parser is not certain about.
- **An explicit commit point.** Once the first byte is written, the status and - **An explicit commit point.** Once the first byte is written, the status and
headers are gone and no `after` middleware can change them. That is a real headers are gone and no `after` middleware can change them. That is a real
semantic change to the middleware contract and it has to be stated, not semantic change to the middleware contract and it has to be stated, not
discovered — the `after` chain currently runs on *every* response and discovered — the `after` chain currently runs on *every* response and security
security headers depend on it. headers depend on it.
## Decisions locked (brainstorm 2026-09-06)
1. **A separate `StreamHandler` interface and a parallel dispatch path — the
`Resp` path is untouched.** A streaming route registers a `StreamHandler`
(`app.stream(pattern, …)`), distinct from `Handler`. Its output is a stream
outcome carrying status, a header set, and a `BodyProducer` — an interface
(a class with a `next() -> ?Bytes` method, `nil` = done), so the no-closures
doctrine holds. Dispatch resolves a streaming route on its own branch;
`serve_conn` gains one branch: a stream outcome writes its headers then pumps
the producer as chunks, everything else serializes exactly as today. The two
paths coexist, which is what keeps every existing response byte-identical.
2. **Streaming routes opt out of the after-chain, and the framework refuses at
registration to combine one with header-mutating middleware.** The after-chain
is *not* re-plumbed onto the streaming branch. Instead, at startup the
framework raises a loud error naming the conflict if a streaming route is in
the scope of a header-mutating `after` middleware (`SecurityHeaders`, `Cors`)
— never a silent partial application, which the study calls the one
unacceptable answer. A stream handler that wants those headers stamps them
itself via a `security_headers()` helper before committing, so opting out of
the chain does not mean losing them — it means setting them explicitly. Once
`serve_conn` writes the header set, it is frozen; a later mutation attempt
hits a guard that traps or logs in development.
3. **Chunked *request*-body parsing is split into its own future iteration; the
deliberate refusal stays until then.** Inbound parsing is orthogonal to
outbound streaming (a different file, `parse.wo`), it is the single riskiest
security surface in the framework (smuggling), and this slice is already the
largest — so it ships outbound-only. `internal/parse.wo`'s refusal of chunked
request bodies remains the behaviour, correct as it stands.
## Phases ## Phases
### Phase A — the writer seam ### Phase A — the writer seam
- Decide the shape (fork 1) and introduce a way for a handler to emit body - Introduce the `StreamHandler` and `BodyProducer` interfaces and the
bytes progressively instead of returning a complete `Resp`. Handlers are `app.stream` registration (decision 1). A streaming handler emits body bytes
classes, so this is a second interface beside `Handler`, not a callback. progressively through the producer instead of returning a complete `Resp`.
- Keep the existing whole-response path as the default and unchanged: the - Keep the existing whole-response path as the default and unchanged: the
overwhelming majority of responses are small and should not pay for this. overwhelming majority of responses are small and must not pay for this. The
- Verify: the two paths coexist; every existing response is byte-identical. `Resp` path, `serialize()`, `route_req` and the after-chain are untouched.
- Verify: the two paths coexist; every existing response is byte-identical, both
serving gates unchanged.
### Phase B — chunked responses ### Phase B — chunked responses
- Chunk framing, the terminating zero-length chunk, and the interaction with - Chunk framing (hex size, CRLF, bytes, CRLF), the terminating zero-length
keep-alive — a connection whose chunked response was truncated must be closed, chunk, and `Transfer-Encoding: chunked` in the header set; `Content-Length`
not reused. and chunked are mutually exclusive and the stream path emits chunked and never
- `Content-Length` and chunked are mutually exclusive; `serialize()` must pick a length — mutual exclusion holds by construction because it is a separate path
one and never emit both. from `serialize()`.
- `HEAD` on a streaming route: headers only, and decide what `Content-Length` - Keep-alive interaction: a chunked response that completes (zero chunk sent)
claims when the length is unknown. may keep the connection; one truncated by a mid-stream producer trap must close
it, never reuse it, and leave no half-frame.
- `HEAD` on a streaming route: the header set (including `Transfer-Encoding:
chunked`) with no body and no `Content-Length`, since the length is unknown.
- Verify: a chunked response reassembles byte-exactly; a mid-stream trap closes - Verify: a chunked response reassembles byte-exactly; a mid-stream trap closes
the connection rather than leaving a half-frame; `HEAD` is coherent. the connection rather than leaving a half-frame; `HEAD` is coherent.
### Phase C — the commit point and the middleware contract ### Phase C — the commit point and the middleware contract
- Define and enforce when headers are locked. An `after` middleware that tries - Implement the opt-out and the registration refusal (decision 2): a streaming
to mutate a committed response must fail loudly in development rather than route under a header-mutating `after` middleware is refused at startup with a
silently doing nothing. message naming the conflict.
- Decide what happens to `SecurityHeaders` and `Cors` — both are `after` - Provide the `security_headers()` helper so a stream handler stamps the standard
middleware and both must still apply to streamed responses, which means they security headers into its own header set before committing.
have to run *before* the commit for those routes. - Enforce the commit point: after `serve_conn` writes the header set, it is
- Verify: security headers and CORS are present on a streamed response; a frozen; a mutation attempt traps or logs in development rather than silently
post-commit mutation attempt is reported. doing nothing.
- Verify: security headers and CORS are present on a streamed response that asks
for them via the helper; a streaming route wrongly combined with
header-mutating middleware is refused at startup; a post-commit mutation
attempt is reported.
### Phase D — chunked request bodies ### Phase D — the gate and the ledger
- Only if phase C is clean. Parse chunked request bodies with the smuggling
cases enumerated and tested: both `Content-Length` and `Transfer-Encoding`
present, duplicated `Transfer-Encoding`, unknown transfer codings, and
oversized or malformed chunk sizes.
- Every ambiguous case stays a 400-and-close, matching the existing duplicate
`Content-Length` discipline.
- Verify: a well-formed chunked upload arrives intact; every enumerated
smuggling shape is refused.
### Phase E — the gate and the ledger
- A streaming route in a sample, gated on both consumers, plus a large-body leg - A streaming route in a sample, gated on both consumers, plus a large-body leg
proving memory does not scale with response size. proving peak memory does not scale with response size, and a regression check
that the WebSocket 101 hijack path still works untouched.
- Retire the README's parked streaming rows; record the commit-point semantics - Retire the README's parked streaming rows; record the commit-point semantics
where a handler author will find them. and the streaming opt-out where a handler author will find them.
- Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean. - Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean.
## Acceptance Criteria ## Acceptance Criteria
@ -103,54 +131,56 @@ readiness: refine
- **Given** a handler that traps mid-stream, **when** the failure occurs, - **Given** a handler that traps mid-stream, **when** the failure occurs,
**then** the connection is closed rather than reused, no half-frame is left **then** the connection is closed rather than reused, no half-frame is left
behind, and the server survives. behind, and the server survives.
- **Given** a streamed response, **when** it leaves, **then** the security and - **Given** a streamed response whose handler calls `security_headers()`,
CORS headers the `after` chain contributes are still present. **when** it leaves, **then** those headers are present.
- **Given** an `after` middleware attempting to change a committed response, - **Given** a streaming route registered in the scope of a header-mutating
`after` middleware, **when** the app starts, **then** it is refused with a
message naming the conflict — never a silently unheadered stream.
- **Given** an `after` or handler attempting to change a committed response,
**when** it runs, **then** the attempt is reported rather than silently **when** it runs, **then** the attempt is reported rather than silently
dropped. dropped.
- **Given** a `HEAD` request to a streaming route, **when** it is answered, - **Given** a `HEAD` request to a streaming route, **when** it is answered,
**then** the headers are coherent and no body is sent. **then** the headers are coherent (chunked, no `Content-Length`) and no body is
- **Given** a request with both `Content-Length` and `Transfer-Encoding`, sent.
**when** it is parsed, **then** it is refused with 400 and the connection is - **Given** the WebSocket 101 hijack path, **when** an upgrade is handled after
closed — smuggling is refused, never guessed at. this iteration, **then** it still bypasses `serialize()` and works untouched.
- **Given** every pre-existing non-streaming response, **when** both serving - **Given** every pre-existing non-streaming response, **when** both serving
gates run, **then** output is byte-identical to before this iteration. gates run, **then** output is byte-identical to before this iteration.
## Out Of Scope ## Out Of Scope
- **Chunked *request* bodies** — split into their own future iteration
(decision 3). The `internal/parse.wo` refusal stays until then; when it is
picked up it must enumerate the smuggling cases (both `Content-Length` and
`Transfer-Encoding` present, duplicated `Transfer-Encoding`, unknown transfer
codings, oversized or malformed chunk sizes), each a 400-and-close, matching
the existing duplicate `Content-Length` discipline.
- **SSE** — iteration [7](07-sse-and-compression.md), the first consumer. - **SSE** — iteration [7](07-sse-and-compression.md), the first consumer.
- **Compression** — also [7](07-sse-and-compression.md); it composes with - **Compression** — also [7](07-sse-and-compression.md); it composes with
chunking and should not be entangled with building it. chunking and should not be entangled with building it.
- **Byte ranges and `SendFile`** — iteration - **Byte ranges and `SendFile`** — iteration [8](08-static-and-lifecycle.md).
[8](08-static-and-lifecycle.md).
- **Request-body backpressure as a general mechanism.** Reading a body slowly to - **Request-body backpressure as a general mechanism.** Reading a body slowly to
push back on a producer wants cancellation, which porch does not have and push back on a producer wants cancellation, which porch does not have and which
which language iteration 31's actor lifecycle owns. This iteration streams language iteration 31's actor lifecycle owns. This iteration streams *out*; it
*out* and parses chunked *in*; it does not add flow control. does not add flow control. A slow client simply parks the writing fiber.
- **WebSockets.** Already shipped (`ws_accept`, `wsframe`) and deliberately a - **WebSockets.** Already shipped (`ws_accept`, `wsframe`) and deliberately a
hijack that bypasses `serialize()` — that path must keep working untouched, hijack that bypasses `serialize()` — that path must keep working untouched,
which is worth an explicit regression check. which is why phase D checks it.
- **HTTP/2.** Proxy-terminated by doctrine, and parked behind language - **HTTP/2.** Proxy-terminated by doctrine, and parked behind language
iteration 23 regardless. iteration 23 regardless.
## Info ## Info
Forks the spec must settle: No language enhancement is needed: the writer seam is `net.write` (id 54) called
repeatedly for framing, a `BodyProducer` sourcing bytes from `fs.read_at`
(id 44) or an actor `receive`, and interfaces/classes for the producer — all
present. The risk in this slice is design risk (the commit contract, framing
correctness, keep-alive on truncation), not a runtime gap, and the streaming
writes ride the existing fiber-per-connection loop, not the per-key actor pool,
so this iteration is not exposed to the lang-41 hang.
1. **What is the writer?** Candidates: a second interface whose method is Forks are settled above. The one the brainstorm moved most was fork 1: rather
called repeatedly until it signals done; a `Resp` variant carrying a producer than folding a producer into `Resp` (which would have made the after-chain apply
object instead of a `Text` body; or a handler that receives the connection and for free), the decision is a genuinely separate `StreamHandler` path — which is
writes directly (which is what `ws_accept` already does via the 101 hijack why fork 2 had to be answered explicitly, and why streaming routes opt out of
sentinel). The third is the least new machinery and the most footgun. The the after-chain with a loud registration refusal rather than inheriting it.
first fits the no-closures doctrine best, since a producer is just another
class with fields.
2. **Does the `after` chain still run for streamed responses?** It must, or
security headers regress. But it cannot run *after* the body. So either
`after` runs at commit time for streaming routes, or streaming routes declare
they opt out and the framework refuses to combine them with header-mutating
middleware. Silent partial application is the one unacceptable answer.
3. **Is chunked request parsing in this iteration at all?** It is separable and
it is the riskiest security surface in the framework. Splitting phase D into
its own iteration is a legitimate outcome of the brainstorm — the study is
explicit that the current refusal is *correct*, so there is no pressure to
rush it.