docs(porch-streaming): brainstorm story 6 (streaming core) to ready
- re-scoped to OUTBOUND streaming only - three decisions: separate StreamHandler/BodyProducer parallel path (Resp path untouched -> existing responses byte-identical); streaming routes opt out of the after-chain, framework refuses at registration to combine with header-mutating middleware (loud, never silent), security_headers() helper lets handlers stamp them; chunked REQUEST bodies split into their own future iteration (parse.wo refusal stays, smuggling cases enumerated for later) - no language enhancement (net.write framing, fs.read_at/actor source, interfaces for producer); rides the fiber loop not the actor pool, so not lang-41-exposed - fixed title inconsistency: "three iterations wait on" -> "two" (7 and 8) - validated against .dev/reference/fiber + the app.wo/serve.wo pipeline. Board synced (cherry picked from commit 15205408e03c3c02a38e00e5d2017a8a11f62f28)
This commit is contained in:
parent
6b820fdd5f
commit
641703903c
2 changed files with 110 additions and 80 deletions
|
|
@ -1195,9 +1195,9 @@ the language arc as v1 history.
|
|||
### ▸ porch — the web framework track
|
||||
|
||||
New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md).
|
||||
Supersedes language iteration 39, now a pointer. **Stories 2, 3, 4 and 5 are
|
||||
`ready` (brainstormed 2026-09-06, forks locked, validated against
|
||||
`.dev/reference/fiber`); 6–8 remain `refine`.** Ordered by dependency; the
|
||||
Supersedes language iteration 39, now a pointer. **Stories 2–6 are `ready`
|
||||
(brainstormed 2026-09-06, forks locked, validated against
|
||||
`.dev/reference/fiber`); 7–8 remain `refine`.** Ordered by dependency; the
|
||||
first slice is deliberately the cheapest so the store pattern and gate shape are
|
||||
proven before the runtime and `Resp` are touched.
|
||||
|
||||
|
|
@ -1208,7 +1208,7 @@ proven before the runtime and `Resp` are touched.
|
|||
| 3 | [Sessions](porch/03-sessions.md) | ✅ **`ready` 2026-09-06** — after 2. Six decisions locked: row is a pure auth primitive (id/principal/created_at/last_seen, no payload bag); **wall-clock `time.now`, not monotonic `time.ticks`** (sessions survive restart); rotation = login always mints a fresh id (no anon-session model); throttled `last_seen` touch at `idle/20` (not a WAL write per request); `Session` writes `req.principal`; config refuses absolute < idle. Pure `.wo` on iteration 2 + the `@table` engine — no new runtime work |
|
||||
| 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 |
|
||||
| 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) |
|
||||
| 6 | [Streaming core](porch/06-streaming-core.md) | ⬜ the riskiest and highest-leverage slice: incremental writes + chunked framing + an explicit commit point. `serialize()` always emits `Content-Length` today. Chunked REQUEST bodies are deliberately refused (request smuggling) and that refusal must survive |
|
||||
| 6 | [Streaming core](porch/06-streaming-core.md) | ✅ **`ready` 2026-09-06** — riskiest/highest-leverage; **re-scoped to outbound only**. Three decisions: separate `StreamHandler`/`BodyProducer` parallel path (the `Resp` path untouched → existing responses byte-identical); streaming routes **opt out** of the after-chain, framework **refuses at registration** to combine with header-mutating middleware (loud, never silent), handlers stamp headers via a `security_headers()` helper; chunked **REQUEST** bodies **split into their own future iteration** (parse.wo refusal stays). No language enhancement; rides the fiber loop, not the actor pool (not lang-41-exposed) |
|
||||
| 7 | [SSE + compression](porch/07-sse-and-compression.md) | ⬜ after 6. SSE fits the actor/fiber model unusually well; compression carries a real fork — pure-`.wo` DEFLATE (now expressible after iteration 36's bit operators) vs a C builtin. CRC32 finally gets its consumer |
|
||||
| 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip |
|
||||
|
||||
|
|
|
|||
|
|
@ -2,19 +2,25 @@
|
|||
track: porch
|
||||
iteration: "6"
|
||||
status: pending
|
||||
readiness: refine
|
||||
readiness: ready
|
||||
---
|
||||
|
||||
# porch 6 — streaming core: the seam three iterations wait on
|
||||
# porch 6 — streaming core: the seam two iterations wait on
|
||||
|
||||
> Part of [Story — `porch`, the writeonce web framework](00-story.md).
|
||||
> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3.
|
||||
> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3,
|
||||
> re-checked 2026-09-06 against `.dev/reference/fiber` (v3, `3ca9a9d`) and the
|
||||
> existing `internal/serve.wo`/`app.wo` dispatch pipeline.
|
||||
>
|
||||
> The largest and riskiest slice in this track, and the one with the most
|
||||
> downstream value: iterations [7](07-sse-and-compression.md) and
|
||||
> [8](08-static-and-lifecycle.md) are both blocked on it, and porch's README has
|
||||
> carried "lazy body streaming · streaming responses · explicit commit point"
|
||||
> as parked since framework v1.
|
||||
>
|
||||
> Re-scoped by the brainstorm to **outbound streaming only** — chunked *request*
|
||||
> bodies split into their own future iteration (decision 3), so this slice ships
|
||||
> and gates without entangling the smuggling surface.
|
||||
|
||||
## Goals
|
||||
|
||||
|
|
@ -27,69 +33,91 @@ readiness: refine
|
|||
- **Chunked transfer-encoding on the way out**, correctly framed and correctly
|
||||
terminated, because a truncated chunked response is indistinguishable from a
|
||||
network failure to the client and corrupts keep-alive for the connection.
|
||||
- **Chunked request bodies on the way in — carefully.**
|
||||
`internal/parse.wo` **deliberately refuses** them today, with a correct note
|
||||
that silently treating a chunked request as body-less is request smuggling.
|
||||
That refusal is good engineering. It may only be lifted by an implementation
|
||||
that handles the smuggling cases explicitly, and the refusal must remain the
|
||||
behaviour for anything the parser is not certain about.
|
||||
- **An explicit commit point.** Once the first byte is written, the status and
|
||||
headers are gone and no `after` middleware can change them. That is a real
|
||||
semantic change to the middleware contract and it has to be stated, not
|
||||
discovered — the `after` chain currently runs on *every* response and
|
||||
security headers depend on it.
|
||||
discovered — the `after` chain currently runs on *every* response and security
|
||||
headers depend on it.
|
||||
|
||||
## Decisions locked (brainstorm 2026-09-06)
|
||||
|
||||
1. **A separate `StreamHandler` interface and a parallel dispatch path — the
|
||||
`Resp` path is untouched.** A streaming route registers a `StreamHandler`
|
||||
(`app.stream(pattern, …)`), distinct from `Handler`. Its output is a stream
|
||||
outcome carrying status, a header set, and a `BodyProducer` — an interface
|
||||
(a class with a `next() -> ?Bytes` method, `nil` = done), so the no-closures
|
||||
doctrine holds. Dispatch resolves a streaming route on its own branch;
|
||||
`serve_conn` gains one branch: a stream outcome writes its headers then pumps
|
||||
the producer as chunks, everything else serializes exactly as today. The two
|
||||
paths coexist, which is what keeps every existing response byte-identical.
|
||||
2. **Streaming routes opt out of the after-chain, and the framework refuses at
|
||||
registration to combine one with header-mutating middleware.** The after-chain
|
||||
is *not* re-plumbed onto the streaming branch. Instead, at startup the
|
||||
framework raises a loud error naming the conflict if a streaming route is in
|
||||
the scope of a header-mutating `after` middleware (`SecurityHeaders`, `Cors`)
|
||||
— never a silent partial application, which the study calls the one
|
||||
unacceptable answer. A stream handler that wants those headers stamps them
|
||||
itself via a `security_headers()` helper before committing, so opting out of
|
||||
the chain does not mean losing them — it means setting them explicitly. Once
|
||||
`serve_conn` writes the header set, it is frozen; a later mutation attempt
|
||||
hits a guard that traps or logs in development.
|
||||
3. **Chunked *request*-body parsing is split into its own future iteration; the
|
||||
deliberate refusal stays until then.** Inbound parsing is orthogonal to
|
||||
outbound streaming (a different file, `parse.wo`), it is the single riskiest
|
||||
security surface in the framework (smuggling), and this slice is already the
|
||||
largest — so it ships outbound-only. `internal/parse.wo`'s refusal of chunked
|
||||
request bodies remains the behaviour, correct as it stands.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase A — the writer seam
|
||||
|
||||
- Decide the shape (fork 1) and introduce a way for a handler to emit body
|
||||
bytes progressively instead of returning a complete `Resp`. Handlers are
|
||||
classes, so this is a second interface beside `Handler`, not a callback.
|
||||
- Introduce the `StreamHandler` and `BodyProducer` interfaces and the
|
||||
`app.stream` registration (decision 1). A streaming handler emits body bytes
|
||||
progressively through the producer instead of returning a complete `Resp`.
|
||||
- Keep the existing whole-response path as the default and unchanged: the
|
||||
overwhelming majority of responses are small and should not pay for this.
|
||||
- Verify: the two paths coexist; every existing response is byte-identical.
|
||||
overwhelming majority of responses are small and must not pay for this. The
|
||||
`Resp` path, `serialize()`, `route_req` and the after-chain are untouched.
|
||||
- Verify: the two paths coexist; every existing response is byte-identical, both
|
||||
serving gates unchanged.
|
||||
|
||||
### Phase B — chunked responses
|
||||
|
||||
- Chunk framing, the terminating zero-length chunk, and the interaction with
|
||||
keep-alive — a connection whose chunked response was truncated must be closed,
|
||||
not reused.
|
||||
- `Content-Length` and chunked are mutually exclusive; `serialize()` must pick
|
||||
one and never emit both.
|
||||
- `HEAD` on a streaming route: headers only, and decide what `Content-Length`
|
||||
claims when the length is unknown.
|
||||
- Chunk framing (hex size, CRLF, bytes, CRLF), the terminating zero-length
|
||||
chunk, and `Transfer-Encoding: chunked` in the header set; `Content-Length`
|
||||
and chunked are mutually exclusive and the stream path emits chunked and never
|
||||
a length — mutual exclusion holds by construction because it is a separate path
|
||||
from `serialize()`.
|
||||
- Keep-alive interaction: a chunked response that completes (zero chunk sent)
|
||||
may keep the connection; one truncated by a mid-stream producer trap must close
|
||||
it, never reuse it, and leave no half-frame.
|
||||
- `HEAD` on a streaming route: the header set (including `Transfer-Encoding:
|
||||
chunked`) with no body and no `Content-Length`, since the length is unknown.
|
||||
- Verify: a chunked response reassembles byte-exactly; a mid-stream trap closes
|
||||
the connection rather than leaving a half-frame; `HEAD` is coherent.
|
||||
|
||||
### Phase C — the commit point and the middleware contract
|
||||
|
||||
- Define and enforce when headers are locked. An `after` middleware that tries
|
||||
to mutate a committed response must fail loudly in development rather than
|
||||
silently doing nothing.
|
||||
- Decide what happens to `SecurityHeaders` and `Cors` — both are `after`
|
||||
middleware and both must still apply to streamed responses, which means they
|
||||
have to run *before* the commit for those routes.
|
||||
- Verify: security headers and CORS are present on a streamed response; a
|
||||
post-commit mutation attempt is reported.
|
||||
- Implement the opt-out and the registration refusal (decision 2): a streaming
|
||||
route under a header-mutating `after` middleware is refused at startup with a
|
||||
message naming the conflict.
|
||||
- Provide the `security_headers()` helper so a stream handler stamps the standard
|
||||
security headers into its own header set before committing.
|
||||
- Enforce the commit point: after `serve_conn` writes the header set, it is
|
||||
frozen; a mutation attempt traps or logs in development rather than silently
|
||||
doing nothing.
|
||||
- Verify: security headers and CORS are present on a streamed response that asks
|
||||
for them via the helper; a streaming route wrongly combined with
|
||||
header-mutating middleware is refused at startup; a post-commit mutation
|
||||
attempt is reported.
|
||||
|
||||
### Phase D — chunked request bodies
|
||||
|
||||
- Only if phase C is clean. Parse chunked request bodies with the smuggling
|
||||
cases enumerated and tested: both `Content-Length` and `Transfer-Encoding`
|
||||
present, duplicated `Transfer-Encoding`, unknown transfer codings, and
|
||||
oversized or malformed chunk sizes.
|
||||
- Every ambiguous case stays a 400-and-close, matching the existing duplicate
|
||||
`Content-Length` discipline.
|
||||
- Verify: a well-formed chunked upload arrives intact; every enumerated
|
||||
smuggling shape is refused.
|
||||
|
||||
### Phase E — the gate and the ledger
|
||||
### Phase D — the gate and the ledger
|
||||
|
||||
- A streaming route in a sample, gated on both consumers, plus a large-body leg
|
||||
proving memory does not scale with response size.
|
||||
proving peak memory does not scale with response size, and a regression check
|
||||
that the WebSocket 101 hijack path still works untouched.
|
||||
- Retire the README's parked streaming rows; record the commit-point semantics
|
||||
where a handler author will find them.
|
||||
and the streaming opt-out where a handler author will find them.
|
||||
- Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
|
@ -103,54 +131,56 @@ readiness: refine
|
|||
- **Given** a handler that traps mid-stream, **when** the failure occurs,
|
||||
**then** the connection is closed rather than reused, no half-frame is left
|
||||
behind, and the server survives.
|
||||
- **Given** a streamed response, **when** it leaves, **then** the security and
|
||||
CORS headers the `after` chain contributes are still present.
|
||||
- **Given** an `after` middleware attempting to change a committed response,
|
||||
- **Given** a streamed response whose handler calls `security_headers()`,
|
||||
**when** it leaves, **then** those headers are present.
|
||||
- **Given** a streaming route registered in the scope of a header-mutating
|
||||
`after` middleware, **when** the app starts, **then** it is refused with a
|
||||
message naming the conflict — never a silently unheadered stream.
|
||||
- **Given** an `after` or handler attempting to change a committed response,
|
||||
**when** it runs, **then** the attempt is reported rather than silently
|
||||
dropped.
|
||||
- **Given** a `HEAD` request to a streaming route, **when** it is answered,
|
||||
**then** the headers are coherent and no body is sent.
|
||||
- **Given** a request with both `Content-Length` and `Transfer-Encoding`,
|
||||
**when** it is parsed, **then** it is refused with 400 and the connection is
|
||||
closed — smuggling is refused, never guessed at.
|
||||
**then** the headers are coherent (chunked, no `Content-Length`) and no body is
|
||||
sent.
|
||||
- **Given** the WebSocket 101 hijack path, **when** an upgrade is handled after
|
||||
this iteration, **then** it still bypasses `serialize()` and works untouched.
|
||||
- **Given** every pre-existing non-streaming response, **when** both serving
|
||||
gates run, **then** output is byte-identical to before this iteration.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- **Chunked *request* bodies** — split into their own future iteration
|
||||
(decision 3). The `internal/parse.wo` refusal stays until then; when it is
|
||||
picked up it must enumerate the smuggling cases (both `Content-Length` and
|
||||
`Transfer-Encoding` present, duplicated `Transfer-Encoding`, unknown transfer
|
||||
codings, oversized or malformed chunk sizes), each a 400-and-close, matching
|
||||
the existing duplicate `Content-Length` discipline.
|
||||
- **SSE** — iteration [7](07-sse-and-compression.md), the first consumer.
|
||||
- **Compression** — also [7](07-sse-and-compression.md); it composes with
|
||||
chunking and should not be entangled with building it.
|
||||
- **Byte ranges and `SendFile`** — iteration
|
||||
[8](08-static-and-lifecycle.md).
|
||||
- **Byte ranges and `SendFile`** — iteration [8](08-static-and-lifecycle.md).
|
||||
- **Request-body backpressure as a general mechanism.** Reading a body slowly to
|
||||
push back on a producer wants cancellation, which porch does not have and
|
||||
which language iteration 31's actor lifecycle owns. This iteration streams
|
||||
*out* and parses chunked *in*; it does not add flow control.
|
||||
push back on a producer wants cancellation, which porch does not have and which
|
||||
language iteration 31's actor lifecycle owns. This iteration streams *out*; it
|
||||
does not add flow control. A slow client simply parks the writing fiber.
|
||||
- **WebSockets.** Already shipped (`ws_accept`, `wsframe`) and deliberately a
|
||||
hijack that bypasses `serialize()` — that path must keep working untouched,
|
||||
which is worth an explicit regression check.
|
||||
which is why phase D checks it.
|
||||
- **HTTP/2.** Proxy-terminated by doctrine, and parked behind language
|
||||
iteration 23 regardless.
|
||||
|
||||
## Info
|
||||
|
||||
Forks the spec must settle:
|
||||
No language enhancement is needed: the writer seam is `net.write` (id 54) called
|
||||
repeatedly for framing, a `BodyProducer` sourcing bytes from `fs.read_at`
|
||||
(id 44) or an actor `receive`, and interfaces/classes for the producer — all
|
||||
present. The risk in this slice is design risk (the commit contract, framing
|
||||
correctness, keep-alive on truncation), not a runtime gap, and the streaming
|
||||
writes ride the existing fiber-per-connection loop, not the per-key actor pool,
|
||||
so this iteration is not exposed to the lang-41 hang.
|
||||
|
||||
1. **What is the writer?** Candidates: a second interface whose method is
|
||||
called repeatedly until it signals done; a `Resp` variant carrying a producer
|
||||
object instead of a `Text` body; or a handler that receives the connection and
|
||||
writes directly (which is what `ws_accept` already does via the 101 hijack
|
||||
sentinel). The third is the least new machinery and the most footgun. The
|
||||
first fits the no-closures doctrine best, since a producer is just another
|
||||
class with fields.
|
||||
2. **Does the `after` chain still run for streamed responses?** It must, or
|
||||
security headers regress. But it cannot run *after* the body. So either
|
||||
`after` runs at commit time for streaming routes, or streaming routes declare
|
||||
they opt out and the framework refuses to combine them with header-mutating
|
||||
middleware. Silent partial application is the one unacceptable answer.
|
||||
3. **Is chunked request parsing in this iteration at all?** It is separable and
|
||||
it is the riskiest security surface in the framework. Splitting phase D into
|
||||
its own iteration is a legitimate outcome of the brainstorm — the study is
|
||||
explicit that the current refusal is *correct*, so there is no pressure to
|
||||
rush it.
|
||||
Forks are settled above. The one the brainstorm moved most was fork 1: rather
|
||||
than folding a producer into `Resp` (which would have made the after-chain apply
|
||||
for free), the decision is a genuinely separate `StreamHandler` path — which is
|
||||
why fork 2 had to be answered explicitly, and why streaming routes opt out of
|
||||
the after-chain with a loud registration refusal rather than inheriting it.
|
||||
|
|
|
|||
Loading…
Reference in a new issue