docs(rv2-obs): rv2 7 observability brainstormed to ready

- four forks settled with KISS defaults grounded in runtime/src: counters+gauges only (profiling split out), Prometheus text rendered in .wo from a map<Text, Int>, pull via proc.metrics(), stack trace on trap lands first
- phases A (trace on trap at both trap sites) / B (proc.metrics from existing gc/arena/fiber fields) / C (porch mounts /metrics — consumer's phase)
- builtin id to be confirmed against WO_B_MAX at build time (random_bytes claims 119 per porch 2's brief)
- review_pending marker: forks auto-approved 2026-09-09, developer second review before code lands
- board row: refine -> ready

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit feb11c3aad613ed7f41b280b27c1f6c0dda92ec7)
This commit is contained in:
shoney.arickathil 2026-09-09 16:25:50 +02:00
parent ed45ac7c06
commit e1b9ada190
2 changed files with 92 additions and 42 deletions

View file

@ -1611,7 +1611,7 @@ starts. Edges in [dependency graph section 6](../00-dependency-graph.md).
| 4 | [termios adoption](runtime-v2/04-termios.md) | ✅ **DONE 2026-09-02** — `term.raw/restore`; restore proven a runtime obligation twice (DIV0 while raw, and the double-raw refusal itself) | | 4 | [termios adoption](runtime-v2/04-termios.md) | ✅ **DONE 2026-09-02** — `term.raw/restore`; restore proven a runtime obligation twice (DIV0 while raw, and the double-raw refusal itself) |
| 5 | [fd passing](runtime-v2/05-fd-passing.md) | ✅ **DONE 2026-09-02** — `net.send_fd`/`recv_fd`/`connect_unix`; a tty crossed the socket, was raw'd through the received copy and restored at destroy — the wmux handover in miniature | | 5 | [fd passing](runtime-v2/05-fd-passing.md) | ✅ **DONE 2026-09-02** — `net.send_fd`/`recv_fd`/`connect_unix`; a tty crossed the socket, was raw'd through the received copy and restored at destroy — the wmux handover in miniature |
| 6 | [term.size + term.width](runtime-v2/06-term-size-width.md) | ✅ **DONE 2026-09-02** — TIOCGWINSZ read twin (nil = not a tty) and libc wcwidth under C.UTF-8; the only runtime work the whole wmux parity ladder needs | | 6 | [term.size + term.width](runtime-v2/06-term-size-width.md) | ✅ **DONE 2026-09-02** — TIOCGWINSZ read twin (nil = not a tty) and libc wcwidth under C.UTF-8; the only runtime work the whole wmux parity ladder needs |
| 7 | [observability](runtime-v2/07-observability.md) | ⬜ `refine` — **moved here 2026-09-06** from language iteration 30 (`was_language_iteration: 30`). Runtime metrics/gauges, a `pprof`-equivalent profile, stack-trace-on-trap; consumers named (porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry). Forks: counters-only vs profiling, exposition format, pull vs push, trace-on-trap as a separable first slice. Stretches the track's charter (instrumentation, not processes/terminals/signals) — noted in the story | | 7 | [observability](runtime-v2/07-observability.md) | ⬜ **`ready` 2026-09-09** (was `refine`; moved here 2026-09-06 from language iteration 30). Four forks locked, grounded in `runtime/src`: (1) **counters/gauges only** — they already exist as fields (`gc_traced_cnt`, `gc_alloc_bytes`, `gc_step_no`, arena `used`, `nfibers`, `nchildren`, `ntls`); profiling split to its own later iteration; (2) **Prometheus text rendered in `.wo`** from a `map<Text,Int>` (no new record, no JSON); (3) **pull** via `proc.metrics() -> map<Text,Int>`, per shard; push/OTel deferred by name; (4) **stack-trace-on-trap lands first, alone** — the trap path already has method+line (per-method line table) and `vm_unwind` walks frames, so phase A appends `at METHOD line N` per frame under the existing trap line. Phases A trace → B `proc.metrics()` → C porch mounts `/metrics` (porch's). Builtin id confirmed against `WO_B_MAX` at build (shared enum). Consumers: porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry. Forks auto-approved, `review_pending` |
| 8 | [symmetric cipher (AEAD)](runtime-v2/08-symmetric-cipher.md) | 🔄 **in-progress** — the **first rung of the TLS ladder** (gates rv2 9). **Phases A + B + C LANDED 2026-09-08**: A ChaCha20-Poly1305 (ids 111/112, RFC 8439 §2.8.2); B AES-128/256-GCM (ids 113/114) via AES-NI+PCLMULQDQ; C portable constant-time software AES-GCM fallback (S-box via GF-inverse ladder, bit-by-bit GHASH) — AES-GCM now on any CPU, dispatched hw-or-sw. All hand-rolled, constant-time, both AES paths NIST cases 4 & 16 byte-exact, KAT-gated in test_crypto (**48/0**), ASan/UBSan clean. Remaining: D cookie wrapper → E gate (ARMv8 hw path deferred). Consumers: rv2 9 TLS + porch encrypted cookies | | 8 | [symmetric cipher (AEAD)](runtime-v2/08-symmetric-cipher.md) | 🔄 **in-progress** — the **first rung of the TLS ladder** (gates rv2 9). **Phases A + B + C LANDED 2026-09-08**: A ChaCha20-Poly1305 (ids 111/112, RFC 8439 §2.8.2); B AES-128/256-GCM (ids 113/114) via AES-NI+PCLMULQDQ; C portable constant-time software AES-GCM fallback (S-box via GF-inverse ladder, bit-by-bit GHASH) — AES-GCM now on any CPU, dispatched hw-or-sw. All hand-rolled, constant-time, both AES paths NIST cases 4 & 16 byte-exact, KAT-gated in test_crypto (**48/0**), ASan/UBSan clean. Remaining: D cookie wrapper → E gate (ARMv8 hw path deferred). Consumers: rv2 9 TLS + porch encrypted cookies |
| 9 | [in-process TLS](runtime-v2/09-in-process-tls.md) | ✅ **DONE 2026-09-09** — in-process TLS 1.3 **both directions**, **retired the "TLS is the proxy's job" doctrine** (34/38/porch corrected). Hand-rolled, 1.3-only, RSA+ECDSA+full X.509; KAT'd vs **RFC 8448** / real certs, ASan/UBSan clean. **A–E crypto** (AEAD, HKDF, X25519, sign/verify, X.509 + SAN + basicConstraints/EKU) → **F client** (`net.connect_tls`/`read_tls`/`write_tls`, ids 115–117) → **G server** (constant-time RSA-PSS + ECDSA-P256 signing w/ RFC 6979, server FSM, `net.accept_tls` id 118, `wo_pkey_parse`). Live-gated: `just tls` 5/0 (outbound) + `just tls-server` 4/0 (inbound, openssl s_client EC+RSA). test_tls 123/0, test_crypto 130/0, full suite 0 fail. Deferred follow-ups (non-blocking): park-based handshake, `TlsConn` object, connection pooling, close_notify, complete-formula EC ladder. Forks auto-approved 2026-09-08/09, `review_pending`. The project's **highest-risk** work — done | | 9 | [in-process TLS](runtime-v2/09-in-process-tls.md) | ✅ **DONE 2026-09-09** — in-process TLS 1.3 **both directions**, **retired the "TLS is the proxy's job" doctrine** (34/38/porch corrected). Hand-rolled, 1.3-only, RSA+ECDSA+full X.509; KAT'd vs **RFC 8448** / real certs, ASan/UBSan clean. **A–E crypto** (AEAD, HKDF, X25519, sign/verify, X.509 + SAN + basicConstraints/EKU) → **F client** (`net.connect_tls`/`read_tls`/`write_tls`, ids 115–117) → **G server** (constant-time RSA-PSS + ECDSA-P256 signing w/ RFC 6979, server FSM, `net.accept_tls` id 118, `wo_pkey_parse`). Live-gated: `just tls` 5/0 (outbound) + `just tls-server` 4/0 (inbound, openssl s_client EC+RSA). test_tls 123/0, test_crypto 130/0, full suite 0 fail. Deferred follow-ups (non-blocking): park-based handshake, `TlsConn` object, connection pooling, close_notify, complete-formula EC ladder. Forks auto-approved 2026-09-08/09, `review_pending`. The project's **highest-risk** work — done |

View file

@ -3,18 +3,18 @@ track: runtime-v2
iteration: "7" iteration: "7"
was_language_iteration: "30" was_language_iteration: "30"
status: pending status: pending
readiness: refine readiness: ready
review_pending: "forks auto-approved 2026-09-09 for autonomous execution — developer second review before code lands; four forks settled with KISS defaults grounded in runtime/src (existing gauges, existing line table)"
--- ---
# runtime-v2 7 — observability: metrics, profiling, and traces on trap # runtime-v2 7 — observability: trace on trap, then counters and gauges
> Moved 2026-09-06 from the language track (was language iteration 30, the > Moved 2026-09-06 from the language track (was language iteration 30, the
> number a dozen docs still point at) into runtime-v2, whose builtin-sized-seam > number a dozen docs still point at) into runtime-v2, whose builtin-sized-seam
> shape it fits. It stretches the track's original processes/terminals/signals > shape it fits. **Brainstormed to `ready` 2026-09-09**: the four forks below are
> charter — observability is runtime instrumentation of the VM, GC and shards — > settled, grounded in what the runtime already holds rather than in what an
> but the track already grew past its first five seams. **`readiness: refine`** — > observability stack usually ships. Profiling is split out (fork 1) — this
> the gap, its consumers and its forks are named here, nothing is brainstormed to > iteration is the cheap, high-value half.
> `ready` yet.
## Why this exists ## Why this exists
@ -37,53 +37,103 @@ once:
- The **rate limiter**'s ephemeral-row expiry is lazy "because porch has no - The **rate limiter**'s ephemeral-row expiry is lazy "because porch has no
timer and iteration 30 owns" the sweep story (that "iteration 30" is now this timer and iteration 30 owns" the sweep story (that "iteration 30" is now this
one — [porch 1](../porch/01-store-backed-middleware.md)). one — [porch 1](../porch/01-store-backed-middleware.md)).
- **Debugging a trap** today is one stderr line — `trap CODE in METHOD at line
N: MESSAGE` — with no call stack, so a trap three calls deep names only the
innermost frame.
Fiber ships `expvar` and `pprof` as middleware; the equivalents here are runtime Fiber ships `expvar` and `pprof` as middleware; the equivalents here are runtime
work, because the numbers they expose (allocations, fiber counts, shard load, work, because the numbers they expose (allocations, fiber counts, shard load,
GC pauses) live in the C runtime, not in `.wo`. GC pauses) live in the C runtime, not in `.wo`.
## What it should deliver (scope to be refined) ## Decisions locked (2026-09-09)
- **Runtime counters and gauges** — allocations, arena high-water, live fiber and 1. **Counters and gauges only; profiling is its own later iteration.** The
actor counts, per-shard load, GC pause totals, request counters — exposed gauges this iteration exposes already exist as runtime fields —
through one endpoint the app can mount. `gc_traced_cnt`, `gc_alloc_bytes`, `gc_step_no` (obj.h), the arena's `used`,
- **A profiling story** — CPU and heap sampling, the `pprof` equivalent, so a hot `nfibers`, `nchildren`, `ntls` (vm.h) — so exposing them is a read, not new
path can be found rather than guessed at. machinery. CPU/heap profiling needs sampling infrastructure in the VM and is a
- **A stack trace on trap** — today a trap is a 500 and a line; a trace at the different size of commitment; it gets its own runtime-v2 iteration when a
trap site is the cheapest debugging win and may be separable from the metrics consumer measures the need. (The story always allowed this split.)
work. 2. **Exposition is Prometheus text, rendered in `.wo`.** The runtime returns
numbers; the *format* is the consumer's. The builtin hands back a
`map<Text, Int>` (an existing container — no new record class), and porch
renders the ops-standard Prometheus text (`name value` lines) from it in a
few `.wo` lines. No `expvar`-style JSON in this iteration (a `json.encode` of
the same map is a one-liner if a consumer asks); no format negotiation.
3. **Pull, not push.** A `proc.metrics() -> map<Text, Int>` builtin — `proc`
because it is this process/shard's introspection, beside `proc.run`/`spawn` —
and porch mounts `/metrics` on it. Push to a collector is possible now that
`net.connect`/`net.connect_tls` exist, but it needs a collector protocol and a
consumer; deferred by name. The snapshot is **per shard** (the calling
shard's gauges); a cross-shard aggregate would need an inbox round-trip and
is deferred — scrape each shard or sum in the app.
4. **Stack trace on trap lands first, alone, in this iteration.** Highest
debugging value per line and zero dependency on the metrics half. The trap
path already resolves method + line through the per-method line table
(loader.c validates it ascending); `vm_unwind` already walks the frame stack.
Phase A walks the frames *before* unwinding and prints one ` at METHOD line
N` line per frame under the existing trap line — both the main-fiber trap
(main.c) and the fiber trap (vm.c) sites. Stderr only, always on (a trap is
already a stderr event); no new builtin, no format flag.
## Forks the brainstorm must settle Builtin id: the next free after the ones being claimed ahead of it
(`random_bytes` takes 119 per porch 2's brief) — **confirm against `WO_B_MAX`
in `runtime/src/wob.h` at build time; the id space is one shared enum**
(wob.h + emit.ml/types.ml + loader.c arity), the lesson porch 2's brief
recorded.
1. **Counters only, or profiling too?** Counters and gauges are a bounded, mostly ## Phases
`.wo`-plus-a-few-builtins surface; CPU/heap profiling needs sampling
machinery in the runtime and is a much larger commitment. Splitting profiling - **A — stack trace on trap.** At both trap-report sites, walk the trapping
into its own iteration is a legitimate outcome. fiber's frames innermost-first and print ` at <method> line <n>` per frame
2. **Exposition format.** Prometheus text (the ops-standard, scrape-friendly), (line from the method's line table at the frame's pc; `?` when a method has
an `expvar`-style JSON blob, or both. The format decides who can consume it no table). Verify: a corpus/regress fixture that traps three calls deep shows
without a translator. three `at` lines in order under the trap line; a top-level trap shows one;
3. **Pull endpoint or push.** A mounted `/metrics` endpoint (pull) fits the every existing fixture's first stderr line is byte-unchanged (the trace is
single-binary model; a push to a collector needs `net.connect` — which now appended, never prepended).
exists (id 110, landed 2026-09-07; `net.connect_tls` for an HTTPS collector, - **B — `proc.metrics()`.** The builtin fills a `map<Text, Int>` from the
runtime-v2 9) — so push is possible, but pull is still the simpler default; shard's existing fields — at least `arena_used_bytes`, `gc_traced_objects`,
say which. `gc_alloc_bytes_since_cycle`, `gc_slices`, `live_fibers`, `live_children`,
4. **Is stack-trace-on-trap in this iteration at all?** It is separable, it is `live_tls_conns`, plus `shard_id` — registered in wob.h + types.ml + loader.c
the highest debugging value per line, and it touches the trap path rather than (module member, `proc`). Verify: a runtime test asserts every key is present
the metrics path — a candidate to land first and alone. and that `live_fibers` rises with spawned fibers and falls when they finish;
the map's keys are stable names (they are the metric names porch will emit).
- **C — porch mounts `/metrics`** rendering Prometheus text from the map. This is
the **consumer's** phase — pure `.wo`, owned by porch 8's lifecycle slice — and
is named here so the builtin ships with its consumer visible, not built here.
## Acceptance Criteria
- **Given** a program that traps three calls deep, **when** it traps, **then**
stderr carries the existing trap line unchanged followed by three `at METHOD
line N` lines, innermost first.
- **Given** every existing fixture and gate, **when** run, **then** the first
stderr line of each trap is byte-identical to before (the trace only appends).
- **Given** a `.wo` program that spawns N fibers and calls `proc.metrics()`,
**when** inspected, **then** `live_fibers` reflects them and every named key is
present with an `Int`.
- **Given** the map, **when** porch renders it, **then** the output is valid
Prometheus text (one `name value` line per key) — proven in porch's own gate
when phase C lands.
## Out of scope (named, owned elsewhere) ## Out of scope (named, owned elsewhere)
- **Per-change CI and fuzzing.** Frequently lumped under the old "iteration 30" - **CPU/heap profiling** (`pprof`'s equivalent) — its own runtime-v2 iteration
but they are tooling and process, not a runtime surface; they belong to a when a consumer measures the need (fork 1).
CI/ops story, not this one. The benchmark *harness* already exists (language - **Push / OpenTelemetry export** — needs a collector protocol and a consumer;
iteration 22). `net.connect_tls` now exists, so the transport is no longer the blocker.
- **Distributed tracing / OpenTelemetry export.** Needs `net.connect` - **Cross-shard aggregation** of the gauges — an inbox round-trip; scrape or sum
(iteration 38) and a wire protocol; a later slice if a consumer appears. in the app for now.
- **`expvar`-style JSON**, format flags, per-request latency histograms — later,
on demand.
- **Per-change CI and fuzzing.** Tooling and process, not a runtime surface;
the benchmark *harness* already exists (language iteration 22).
- **Alerting, dashboards.** Downstream of exposition, not the runtime's job. - **Alerting, dashboards.** Downstream of exposition, not the runtime's job.
## Info ## Info
Consumers exist and are named above, so this is not a primitive shipped as Consumers exist and are named above, so this is not a primitive shipped as
decoration. Ordering: stack-trace-on-trap has no dependency and could lead; decoration. Ordering: phase A (trace on trap) has no dependency and leads;
counters/gauges are next; profiling is the heaviest and most separable. Nothing phase B (counters) follows; phase C is porch's. Nothing here depends on the
here depends on the porch track — the dependency runs the other way. porch track — the dependency runs the other way. Small: a frame walk at two
trap sites, one module builtin reading fields that already exist.