shoney/writeonce
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

docs(porch-streaming): brainstorm story 6 (streaming core) to ready

Browse source 
- re-scoped to OUTBOUND streaming only
- three decisions: separate StreamHandler/BodyProducer parallel path (Resp
  path untouched -> existing responses byte-identical); streaming routes opt
  out of the after-chain, framework refuses at registration to combine with
  header-mutating middleware (loud, never silent), security_headers() helper
  lets handlers stamp them; chunked REQUEST bodies split into their own future
  iteration (parse.wo refusal stays, smuggling cases enumerated for later)
- no language enhancement (net.write framing, fs.read_at/actor source,
  interfaces for producer); rides the fiber loop not the actor pool, so not
  lang-41-exposed
- fixed title inconsistency: "three iterations wait on" -> "two" (7 and 8)
- validated against .dev/reference/fiber + the app.wo/serve.wo pipeline. Board synced

(cherry picked from commit 15205408e03c3c02a38e00e5d2017a8a11f62f28)
This commit is contained in:
shoney.arickathil 2026-09-06 17:31:06 +02:00
parent 6b820fdd5f
commit 641703903c
2 changed files with 110 additions and 80 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
docs/stories/00-status.md
View file

@ -1195,9 +1195,9 @@ the language arc as v1 history.
### ▸ porch — the web framework track
New 2026-08-26, from [the Fiber v3.5.0 parity study](../plan/exploration/fiber/00-fiber-parity.md).
Supersedes language iteration 39, now a pointer. **Stories 2, 3, 4 and 5 are
`ready` (brainstormed 2026-09-06, forks locked, validated against
`.dev/reference/fiber`); 6–8 remain `refine`.** Ordered by dependency; the
Supersedes language iteration 39, now a pointer. **Stories 2–6 are `ready`
(brainstormed 2026-09-06, forks locked, validated against
`.dev/reference/fiber`); 7–8 remain `refine`.** Ordered by dependency; the
first slice is deliberately the cheapest so the store pattern and gate shape are
proven before the runtime and `Resp` are touched.
@ -1208,7 +1208,7 @@ proven before the runtime and `Resp` are touched.
| 3 | [Sessions](porch/03-sessions.md) | ✅ **`ready` 2026-09-06** — after 2. Six decisions locked: row is a pure auth primitive (id/principal/created_at/last_seen, no payload bag); **wall-clock `time.now`, not monotonic `time.ticks`** (sessions survive restart); rotation = login always mints a fresh id (no anon-session model); throttled `last_seen` touch at `idle/20` (not a WAL write per request); `Session` writes `req.principal`; config refuses absolute < idle. Pure `.wo` on iteration 2 + the `@table` engine — no new runtime work |
| 4 | [CSRF](porch/04-csrf.md) | ✅ **`ready` 2026-09-06** — after 2 + 3. Five decisions locked: fiber's **hybrid** transport (session-stored `CsrfToken` @table keyed by token + double-submit cookie, both must pass; no CSRF for sessionless apps); opt-in single-use (checkout the example, admin multi-use); a double-click yields a distinct `SPENT` refusal with **no coupling to the lang-41-blocked idempotency**; trusted origin/referer/`Sec-Fetch-Site` as the second layer; refusal classes distinguishable in logs, opaque in body. Plain `@table` CRUD — no actor pool, not blocked on lang-41 |
| 5 | [Routing + response ergonomics](porch/05-routing-response-ergonomics.md) | ✅ **`ready` 2026-09-06** — **independent, any time; NO upstream dependency (not even iteration 2)**. Five decisions: `head` auto-registers with an opt-out (+ `patch`/`options`/`all`); request ids mirror the limiter's trust model with a **non-crypto** source (so no CSPRNG dependency); per-route `body_limit` is a **second check after routing** (global `BODY_MAX` stays the pre-routing ceiling, over-limit = 413); adding `name`/`body_limit` to `Route` is corpus-free; `Vary` accumulates by comma-join. Plus named routes + runtime-checked URL building, q-value ranking (retires the 🔶) |
| 6 | [Streaming core](porch/06-streaming-core.md) | ⬜ the riskiest and highest-leverage slice: incremental writes + chunked framing + an explicit commit point. `serialize()` always emits `Content-Length` today. Chunked REQUEST bodies are deliberately refused (request smuggling) and that refusal must survive |
| 6 | [Streaming core](porch/06-streaming-core.md) | ✅ **`ready` 2026-09-06** — riskiest/highest-leverage; **re-scoped to outbound only**. Three decisions: separate `StreamHandler`/`BodyProducer` parallel path (the `Resp` path untouched → existing responses byte-identical); streaming routes **opt out** of the after-chain, framework **refuses at registration** to combine with header-mutating middleware (loud, never silent), handlers stamp headers via a `security_headers()` helper; chunked **REQUEST** bodies **split into their own future iteration** (parse.wo refusal stays). No language enhancement; rides the fiber loop, not the actor pool (not lang-41-exposed) |
| 7 | [SSE + compression](porch/07-sse-and-compression.md) | ⬜ after 6. SSE fits the actor/fiber model unusually well; compression carries a real fork — pure-`.wo` DEFLATE (now expressible after iteration 36's bit operators) vs a C builtin. CRC32 finally gets its consumer |
| 8 | [Static files + lifecycle](porch/08-static-and-lifecycle.md) | ⬜ static half after 6. Byte ranges, `Last-Modified`/`Cache-Control`, index resolution, listing off-by-default, shutdown hooks (the ledger's "no user teardown hooks yet"), plus healthcheck/favicon/redirect/rewrite/skip |

182
docs/stories/porch/06-streaming-core.md
View file

@ -2,19 +2,25 @@
track: porch
iteration: "6"
status: pending
readiness: refine
readiness: ready
---
# porch 6 — streaming core: the seam three iterations wait on
# porch 6 — streaming core: the seam two iterations wait on
> Part of [Story — `porch`, the writeonce web framework](00-story.md).
> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3.
> Source: [the Fiber parity study](../../plan/exploration/fiber/00-fiber-parity.md) §3,
> re-checked 2026-09-06 against `.dev/reference/fiber` (v3, `3ca9a9d`) and the
> existing `internal/serve.wo`/`app.wo` dispatch pipeline.
>
> The largest and riskiest slice in this track, and the one with the most
> downstream value: iterations [7](07-sse-and-compression.md) and
> [8](08-static-and-lifecycle.md) are both blocked on it, and porch's README has
> carried "lazy body streaming · streaming responses · explicit commit point"
> as parked since framework v1.
>
> Re-scoped by the brainstorm to **outbound streaming only** — chunked *request*
> bodies split into their own future iteration (decision 3), so this slice ships
> and gates without entangling the smuggling surface.
## Goals
@ -27,69 +33,91 @@ readiness: refine
- **Chunked transfer-encoding on the way out**, correctly framed and correctly
terminated, because a truncated chunked response is indistinguishable from a
network failure to the client and corrupts keep-alive for the connection.
- **Chunked request bodies on the way in — carefully.**
`internal/parse.wo` **deliberately refuses** them today, with a correct note
that silently treating a chunked request as body-less is request smuggling.
That refusal is good engineering. It may only be lifted by an implementation
that handles the smuggling cases explicitly, and the refusal must remain the
behaviour for anything the parser is not certain about.
- **An explicit commit point.** Once the first byte is written, the status and
headers are gone and no `after` middleware can change them. That is a real
semantic change to the middleware contract and it has to be stated, not
discovered — the `after` chain currently runs on *every* response and
security headers depend on it.
discovered — the `after` chain currently runs on *every* response and security
headers depend on it.
## Decisions locked (brainstorm 2026-09-06)
1. **A separate `StreamHandler` interface and a parallel dispatch path — the
`Resp` path is untouched.** A streaming route registers a `StreamHandler`
(`app.stream(pattern, …)`), distinct from `Handler`. Its output is a stream
outcome carrying status, a header set, and a `BodyProducer` — an interface
(a class with a `next() -> ?Bytes` method, `nil` = done), so the no-closures
doctrine holds. Dispatch resolves a streaming route on its own branch;
`serve_conn` gains one branch: a stream outcome writes its headers then pumps
the producer as chunks, everything else serializes exactly as today. The two
paths coexist, which is what keeps every existing response byte-identical.
2. **Streaming routes opt out of the after-chain, and the framework refuses at
registration to combine one with header-mutating middleware.** The after-chain
is *not* re-plumbed onto the streaming branch. Instead, at startup the
framework raises a loud error naming the conflict if a streaming route is in
the scope of a header-mutating `after` middleware (`SecurityHeaders`, `Cors`)
— never a silent partial application, which the study calls the one
unacceptable answer. A stream handler that wants those headers stamps them
itself via a `security_headers()` helper before committing, so opting out of
the chain does not mean losing them — it means setting them explicitly. Once
`serve_conn` writes the header set, it is frozen; a later mutation attempt
hits a guard that traps or logs in development.
3. **Chunked *request*-body parsing is split into its own future iteration; the
deliberate refusal stays until then.** Inbound parsing is orthogonal to
outbound streaming (a different file, `parse.wo`), it is the single riskiest
security surface in the framework (smuggling), and this slice is already the
largest — so it ships outbound-only. `internal/parse.wo`'s refusal of chunked
request bodies remains the behaviour, correct as it stands.
## Phases
### Phase A — the writer seam
- Decide the shape (fork 1) and introduce a way for a handler to emit body
bytes progressively instead of returning a complete `Resp`. Handlers are
classes, so this is a second interface beside `Handler`, not a callback.
- Introduce the `StreamHandler` and `BodyProducer` interfaces and the
`app.stream` registration (decision 1). A streaming handler emits body bytes
progressively through the producer instead of returning a complete `Resp`.
- Keep the existing whole-response path as the default and unchanged: the
overwhelming majority of responses are small and should not pay for this.
- Verify: the two paths coexist; every existing response is byte-identical.
overwhelming majority of responses are small and must not pay for this. The
`Resp` path, `serialize()`, `route_req` and the after-chain are untouched.
- Verify: the two paths coexist; every existing response is byte-identical, both
serving gates unchanged.
### Phase B — chunked responses
- Chunk framing, the terminating zero-length chunk, and the interaction with
keep-alive — a connection whose chunked response was truncated must be closed,
not reused.
- `Content-Length` and chunked are mutually exclusive; `serialize()` must pick
one and never emit both.
- `HEAD` on a streaming route: headers only, and decide what `Content-Length`
claims when the length is unknown.
- Chunk framing (hex size, CRLF, bytes, CRLF), the terminating zero-length
chunk, and `Transfer-Encoding: chunked` in the header set; `Content-Length`
and chunked are mutually exclusive and the stream path emits chunked and never
a length — mutual exclusion holds by construction because it is a separate path
from `serialize()`.
- Keep-alive interaction: a chunked response that completes (zero chunk sent)
may keep the connection; one truncated by a mid-stream producer trap must close
it, never reuse it, and leave no half-frame.
- `HEAD` on a streaming route: the header set (including `Transfer-Encoding:
chunked`) with no body and no `Content-Length`, since the length is unknown.
- Verify: a chunked response reassembles byte-exactly; a mid-stream trap closes
the connection rather than leaving a half-frame; `HEAD` is coherent.
### Phase C — the commit point and the middleware contract
- Define and enforce when headers are locked. An `after` middleware that tries
to mutate a committed response must fail loudly in development rather than
silently doing nothing.
- Decide what happens to `SecurityHeaders` and `Cors` — both are `after`
middleware and both must still apply to streamed responses, which means they
have to run *before* the commit for those routes.
- Verify: security headers and CORS are present on a streamed response; a
post-commit mutation attempt is reported.
- Implement the opt-out and the registration refusal (decision 2): a streaming
route under a header-mutating `after` middleware is refused at startup with a
message naming the conflict.
- Provide the `security_headers()` helper so a stream handler stamps the standard
security headers into its own header set before committing.
- Enforce the commit point: after `serve_conn` writes the header set, it is
frozen; a mutation attempt traps or logs in development rather than silently
doing nothing.
- Verify: security headers and CORS are present on a streamed response that asks
for them via the helper; a streaming route wrongly combined with
header-mutating middleware is refused at startup; a post-commit mutation
attempt is reported.
### Phase D — chunked request bodies
- Only if phase C is clean. Parse chunked request bodies with the smuggling
cases enumerated and tested: both `Content-Length` and `Transfer-Encoding`
present, duplicated `Transfer-Encoding`, unknown transfer codings, and
oversized or malformed chunk sizes.
- Every ambiguous case stays a 400-and-close, matching the existing duplicate
`Content-Length` discipline.
- Verify: a well-formed chunked upload arrives intact; every enumerated
smuggling shape is refused.
### Phase E — the gate and the ledger
### Phase D — the gate and the ledger
- A streaming route in a sample, gated on both consumers, plus a large-body leg
proving memory does not scale with response size.
proving peak memory does not scale with response size, and a regression check
that the WebSocket 101 hijack path still works untouched.
- Retire the README's parked streaming rows; record the commit-point semantics
where a handler author will find them.
and the streaming opt-out where a handler author will find them.
- Verify: `just web-app`, `just site`, `just linkcheck` green; ASan clean.
## Acceptance Criteria
@ -103,54 +131,56 @@ readiness: refine
- **Given** a handler that traps mid-stream, **when** the failure occurs,
**then** the connection is closed rather than reused, no half-frame is left
behind, and the server survives.
- **Given** a streamed response, **when** it leaves, **then** the security and
CORS headers the `after` chain contributes are still present.
- **Given** an `after` middleware attempting to change a committed response,
- **Given** a streamed response whose handler calls `security_headers()`,
**when** it leaves, **then** those headers are present.
- **Given** a streaming route registered in the scope of a header-mutating
`after` middleware, **when** the app starts, **then** it is refused with a
message naming the conflict — never a silently unheadered stream.
- **Given** an `after` or handler attempting to change a committed response,
**when** it runs, **then** the attempt is reported rather than silently
dropped.
- **Given** a `HEAD` request to a streaming route, **when** it is answered,
**then** the headers are coherent and no body is sent.
- **Given** a request with both `Content-Length` and `Transfer-Encoding`,
**when** it is parsed, **then** it is refused with 400 and the connection is
closed — smuggling is refused, never guessed at.
**then** the headers are coherent (chunked, no `Content-Length`) and no body is
sent.
- **Given** the WebSocket 101 hijack path, **when** an upgrade is handled after
this iteration, **then** it still bypasses `serialize()` and works untouched.
- **Given** every pre-existing non-streaming response, **when** both serving
gates run, **then** output is byte-identical to before this iteration.
## Out Of Scope
- **Chunked *request* bodies** — split into their own future iteration
(decision 3). The `internal/parse.wo` refusal stays until then; when it is
picked up it must enumerate the smuggling cases (both `Content-Length` and
`Transfer-Encoding` present, duplicated `Transfer-Encoding`, unknown transfer
codings, oversized or malformed chunk sizes), each a 400-and-close, matching
the existing duplicate `Content-Length` discipline.
- **SSE** — iteration [7](07-sse-and-compression.md), the first consumer.
- **Compression** — also [7](07-sse-and-compression.md); it composes with
chunking and should not be entangled with building it.
- **Byte ranges and `SendFile`** — iteration
[8](08-static-and-lifecycle.md).
- **Byte ranges and `SendFile`** — iteration [8](08-static-and-lifecycle.md).
- **Request-body backpressure as a general mechanism.** Reading a body slowly to
push back on a producer wants cancellation, which porch does not have and
which language iteration 31's actor lifecycle owns. This iteration streams
*out* and parses chunked *in*; it does not add flow control.
push back on a producer wants cancellation, which porch does not have and which
language iteration 31's actor lifecycle owns. This iteration streams *out*; it
does not add flow control. A slow client simply parks the writing fiber.
- **WebSockets.** Already shipped (`ws_accept`, `wsframe`) and deliberately a
hijack that bypasses `serialize()` — that path must keep working untouched,
which is worth an explicit regression check.
which is why phase D checks it.
- **HTTP/2.** Proxy-terminated by doctrine, and parked behind language
iteration 23 regardless.
## Info
Forks the spec must settle:
No language enhancement is needed: the writer seam is `net.write` (id 54) called
repeatedly for framing, a `BodyProducer` sourcing bytes from `fs.read_at`
(id 44) or an actor `receive`, and interfaces/classes for the producer — all
present. The risk in this slice is design risk (the commit contract, framing
correctness, keep-alive on truncation), not a runtime gap, and the streaming
writes ride the existing fiber-per-connection loop, not the per-key actor pool,
so this iteration is not exposed to the lang-41 hang.
1. **What is the writer?** Candidates: a second interface whose method is
called repeatedly until it signals done; a `Resp` variant carrying a producer
object instead of a `Text` body; or a handler that receives the connection and
writes directly (which is what `ws_accept` already does via the 101 hijack
sentinel). The third is the least new machinery and the most footgun. The
first fits the no-closures doctrine best, since a producer is just another
class with fields.
2. **Does the `after` chain still run for streamed responses?** It must, or
security headers regress. But it cannot run *after* the body. So either
`after` runs at commit time for streaming routes, or streaming routes declare
they opt out and the framework refuses to combine them with header-mutating
middleware. Silent partial application is the one unacceptable answer.
3. **Is chunked request parsing in this iteration at all?** It is separable and
it is the riskiest security surface in the framework. Splitting phase D into
its own iteration is a legitimate outcome of the brainstorm — the study is
explicit that the current refusal is *correct*, so there is no pressure to
rush it.
Forks are settled above. The one the brainstorm moved most was fork 1: rather
than folding a producer into `Resp` (which would have made the after-chain apply
for free), the decision is a genuinely separate `StreamHandler` path — which is
why fork 2 had to be answered explicitly, and why streaming routes opt out of
the after-chain with a loud registration refusal rather than inheriting it.