From e1b9ada190115c8f0ae30531b90c1ed64c22a741 Mon Sep 17 00:00:00 2001 From: "shoney.arickathil" Date: Wed, 9 Sep 2026 16:25:50 +0200 Subject: [PATCH] docs(rv2-obs): rv2 7 observability brainstormed to ready MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - four forks settled with KISS defaults grounded in runtime/src: counters+gauges only (profiling split out), Prometheus text rendered in .wo from a map, pull via proc.metrics(), stack trace on trap lands first - phases A (trace on trap at both trap sites) / B (proc.metrics from existing gc/arena/fiber fields) / C (porch mounts /metrics — consumer's phase) - builtin id to be confirmed against WO_B_MAX at build time (random_bytes claims 119 per porch 2's brief) - review_pending marker: forks auto-approved 2026-09-09, developer second review before code lands - board row: refine -> ready Co-Authored-By: Claude Fable 5.1 (cherry picked from commit feb11c3aad613ed7f41b280b27c1f6c0dda92ec7) --- docs/stories/00-status.md | 2 +- docs/stories/runtime-v2/07-observability.md | 132 ++++++++++++++------ 2 files changed, 92 insertions(+), 42 deletions(-) diff --git a/docs/stories/00-status.md b/docs/stories/00-status.md index 674b357..16940a2 100644 --- a/docs/stories/00-status.md +++ b/docs/stories/00-status.md @@ -1611,7 +1611,7 @@ starts. Edges in [dependency graph section 6](../00-dependency-graph.md). | 4 | [termios adoption](runtime-v2/04-termios.md) | ✅ **DONE 2026-09-02** — `term.raw/restore`; restore proven a runtime obligation twice (DIV0 while raw, and the double-raw refusal itself) | | 5 | [fd passing](runtime-v2/05-fd-passing.md) | ✅ **DONE 2026-09-02** — `net.send_fd`/`recv_fd`/`connect_unix`; a tty crossed the socket, was raw'd through the received copy and restored at destroy — the wmux handover in miniature | | 6 | [term.size + term.width](runtime-v2/06-term-size-width.md) | ✅ **DONE 2026-09-02** — TIOCGWINSZ read twin (nil = not a tty) and libc wcwidth under C.UTF-8; the only runtime work the whole wmux parity ladder needs | -| 7 | [observability](runtime-v2/07-observability.md) | ⬜ `refine` — **moved here 2026-09-06** from language iteration 30 (`was_language_iteration: 30`). Runtime metrics/gauges, a `pprof`-equivalent profile, stack-trace-on-trap; consumers named (porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry). Forks: counters-only vs profiling, exposition format, pull vs push, trace-on-trap as a separable first slice. Stretches the track's charter (instrumentation, not processes/terminals/signals) — noted in the story | +| 7 | [observability](runtime-v2/07-observability.md) | ⬜ **`ready` 2026-09-09** (was `refine`; moved here 2026-09-06 from language iteration 30). Four forks locked, grounded in `runtime/src`: (1) **counters/gauges only** — they already exist as fields (`gc_traced_cnt`, `gc_alloc_bytes`, `gc_step_no`, arena `used`, `nfibers`, `nchildren`, `ntls`); profiling split to its own later iteration; (2) **Prometheus text rendered in `.wo`** from a `map` (no new record, no JSON); (3) **pull** via `proc.metrics() -> map`, per shard; push/OTel deferred by name; (4) **stack-trace-on-trap lands first, alone** — the trap path already has method+line (per-method line table) and `vm_unwind` walks frames, so phase A appends `at METHOD line N` per frame under the existing trap line. Phases A trace → B `proc.metrics()` → C porch mounts `/metrics` (porch's). Builtin id confirmed against `WO_B_MAX` at build (shared enum). Consumers: porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry. Forks auto-approved, `review_pending` | | 8 | [symmetric cipher (AEAD)](runtime-v2/08-symmetric-cipher.md) | 🔄 **in-progress** — the **first rung of the TLS ladder** (gates rv2 9). **Phases A + B + C LANDED 2026-09-08**: A ChaCha20-Poly1305 (ids 111/112, RFC 8439 §2.8.2); B AES-128/256-GCM (ids 113/114) via AES-NI+PCLMULQDQ; C portable constant-time software AES-GCM fallback (S-box via GF-inverse ladder, bit-by-bit GHASH) — AES-GCM now on any CPU, dispatched hw-or-sw. All hand-rolled, constant-time, both AES paths NIST cases 4 & 16 byte-exact, KAT-gated in test_crypto (**48/0**), ASan/UBSan clean. Remaining: D cookie wrapper → E gate (ARMv8 hw path deferred). Consumers: rv2 9 TLS + porch encrypted cookies | | 9 | [in-process TLS](runtime-v2/09-in-process-tls.md) | ✅ **DONE 2026-09-09** — in-process TLS 1.3 **both directions**, **retired the "TLS is the proxy's job" doctrine** (34/38/porch corrected). Hand-rolled, 1.3-only, RSA+ECDSA+full X.509; KAT'd vs **RFC 8448** / real certs, ASan/UBSan clean. **A–E crypto** (AEAD, HKDF, X25519, sign/verify, X.509 + SAN + basicConstraints/EKU) → **F client** (`net.connect_tls`/`read_tls`/`write_tls`, ids 115–117) → **G server** (constant-time RSA-PSS + ECDSA-P256 signing w/ RFC 6979, server FSM, `net.accept_tls` id 118, `wo_pkey_parse`). Live-gated: `just tls` 5/0 (outbound) + `just tls-server` 4/0 (inbound, openssl s_client EC+RSA). test_tls 123/0, test_crypto 130/0, full suite 0 fail. Deferred follow-ups (non-blocking): park-based handshake, `TlsConn` object, connection pooling, close_notify, complete-formula EC ladder. Forks auto-approved 2026-09-08/09, `review_pending`. The project's **highest-risk** work — done | diff --git a/docs/stories/runtime-v2/07-observability.md b/docs/stories/runtime-v2/07-observability.md index 79563e7..de324c0 100644 --- a/docs/stories/runtime-v2/07-observability.md +++ b/docs/stories/runtime-v2/07-observability.md @@ -3,18 +3,18 @@ track: runtime-v2 iteration: "7" was_language_iteration: "30" status: pending -readiness: refine +readiness: ready +review_pending: "forks auto-approved 2026-09-09 for autonomous execution — developer second review before code lands; four forks settled with KISS defaults grounded in runtime/src (existing gauges, existing line table)" --- -# runtime-v2 7 — observability: metrics, profiling, and traces on trap +# runtime-v2 7 — observability: trace on trap, then counters and gauges > Moved 2026-09-06 from the language track (was language iteration 30, the > number a dozen docs still point at) into runtime-v2, whose builtin-sized-seam -> shape it fits. It stretches the track's original processes/terminals/signals -> charter — observability is runtime instrumentation of the VM, GC and shards — -> but the track already grew past its first five seams. **`readiness: refine`** — -> the gap, its consumers and its forks are named here, nothing is brainstormed to -> `ready` yet. +> shape it fits. **Brainstormed to `ready` 2026-09-09**: the four forks below are +> settled, grounded in what the runtime already holds rather than in what an +> observability stack usually ships. Profiling is split out (fork 1) — this +> iteration is the cheap, high-value half. ## Why this exists @@ -37,53 +37,103 @@ once: - The **rate limiter**'s ephemeral-row expiry is lazy "because porch has no timer and iteration 30 owns" the sweep story (that "iteration 30" is now this one — [porch 1](../porch/01-store-backed-middleware.md)). +- **Debugging a trap** today is one stderr line — `trap CODE in METHOD at line + N: MESSAGE` — with no call stack, so a trap three calls deep names only the + innermost frame. Fiber ships `expvar` and `pprof` as middleware; the equivalents here are runtime work, because the numbers they expose (allocations, fiber counts, shard load, GC pauses) live in the C runtime, not in `.wo`. -## What it should deliver (scope to be refined) +## Decisions locked (2026-09-09) -- **Runtime counters and gauges** — allocations, arena high-water, live fiber and - actor counts, per-shard load, GC pause totals, request counters — exposed - through one endpoint the app can mount. -- **A profiling story** — CPU and heap sampling, the `pprof` equivalent, so a hot - path can be found rather than guessed at. -- **A stack trace on trap** — today a trap is a 500 and a line; a trace at the - trap site is the cheapest debugging win and may be separable from the metrics - work. +1. **Counters and gauges only; profiling is its own later iteration.** The + gauges this iteration exposes already exist as runtime fields — + `gc_traced_cnt`, `gc_alloc_bytes`, `gc_step_no` (obj.h), the arena's `used`, + `nfibers`, `nchildren`, `ntls` (vm.h) — so exposing them is a read, not new + machinery. CPU/heap profiling needs sampling infrastructure in the VM and is a + different size of commitment; it gets its own runtime-v2 iteration when a + consumer measures the need. (The story always allowed this split.) +2. **Exposition is Prometheus text, rendered in `.wo`.** The runtime returns + numbers; the *format* is the consumer's. The builtin hands back a + `map` (an existing container — no new record class), and porch + renders the ops-standard Prometheus text (`name value` lines) from it in a + few `.wo` lines. No `expvar`-style JSON in this iteration (a `json.encode` of + the same map is a one-liner if a consumer asks); no format negotiation. +3. **Pull, not push.** A `proc.metrics() -> map` builtin — `proc` + because it is this process/shard's introspection, beside `proc.run`/`spawn` — + and porch mounts `/metrics` on it. Push to a collector is possible now that + `net.connect`/`net.connect_tls` exist, but it needs a collector protocol and a + consumer; deferred by name. The snapshot is **per shard** (the calling + shard's gauges); a cross-shard aggregate would need an inbox round-trip and + is deferred — scrape each shard or sum in the app. +4. **Stack trace on trap lands first, alone, in this iteration.** Highest + debugging value per line and zero dependency on the metrics half. The trap + path already resolves method + line through the per-method line table + (loader.c validates it ascending); `vm_unwind` already walks the frame stack. + Phase A walks the frames *before* unwinding and prints one ` at METHOD line + N` line per frame under the existing trap line — both the main-fiber trap + (main.c) and the fiber trap (vm.c) sites. Stderr only, always on (a trap is + already a stderr event); no new builtin, no format flag. -## Forks the brainstorm must settle +Builtin id: the next free after the ones being claimed ahead of it +(`random_bytes` takes 119 per porch 2's brief) — **confirm against `WO_B_MAX` +in `runtime/src/wob.h` at build time; the id space is one shared enum** +(wob.h + emit.ml/types.ml + loader.c arity), the lesson porch 2's brief +recorded. -1. **Counters only, or profiling too?** Counters and gauges are a bounded, mostly - `.wo`-plus-a-few-builtins surface; CPU/heap profiling needs sampling - machinery in the runtime and is a much larger commitment. Splitting profiling - into its own iteration is a legitimate outcome. -2. **Exposition format.** Prometheus text (the ops-standard, scrape-friendly), - an `expvar`-style JSON blob, or both. The format decides who can consume it - without a translator. -3. **Pull endpoint or push.** A mounted `/metrics` endpoint (pull) fits the - single-binary model; a push to a collector needs `net.connect` — which now - exists (id 110, landed 2026-09-07; `net.connect_tls` for an HTTPS collector, - runtime-v2 9) — so push is possible, but pull is still the simpler default; - say which. -4. **Is stack-trace-on-trap in this iteration at all?** It is separable, it is - the highest debugging value per line, and it touches the trap path rather than - the metrics path — a candidate to land first and alone. +## Phases + +- **A — stack trace on trap.** At both trap-report sites, walk the trapping + fiber's frames innermost-first and print ` at line ` per frame + (line from the method's line table at the frame's pc; `?` when a method has + no table). Verify: a corpus/regress fixture that traps three calls deep shows + three `at` lines in order under the trap line; a top-level trap shows one; + every existing fixture's first stderr line is byte-unchanged (the trace is + appended, never prepended). +- **B — `proc.metrics()`.** The builtin fills a `map` from the + shard's existing fields — at least `arena_used_bytes`, `gc_traced_objects`, + `gc_alloc_bytes_since_cycle`, `gc_slices`, `live_fibers`, `live_children`, + `live_tls_conns`, plus `shard_id` — registered in wob.h + types.ml + loader.c + (module member, `proc`). Verify: a runtime test asserts every key is present + and that `live_fibers` rises with spawned fibers and falls when they finish; + the map's keys are stable names (they are the metric names porch will emit). +- **C — porch mounts `/metrics`** rendering Prometheus text from the map. This is + the **consumer's** phase — pure `.wo`, owned by porch 8's lifecycle slice — and + is named here so the builtin ships with its consumer visible, not built here. + +## Acceptance Criteria + +- **Given** a program that traps three calls deep, **when** it traps, **then** + stderr carries the existing trap line unchanged followed by three `at METHOD + line N` lines, innermost first. +- **Given** every existing fixture and gate, **when** run, **then** the first + stderr line of each trap is byte-identical to before (the trace only appends). +- **Given** a `.wo` program that spawns N fibers and calls `proc.metrics()`, + **when** inspected, **then** `live_fibers` reflects them and every named key is + present with an `Int`. +- **Given** the map, **when** porch renders it, **then** the output is valid + Prometheus text (one `name value` line per key) — proven in porch's own gate + when phase C lands. ## Out of scope (named, owned elsewhere) -- **Per-change CI and fuzzing.** Frequently lumped under the old "iteration 30" - but they are tooling and process, not a runtime surface; they belong to a - CI/ops story, not this one. The benchmark *harness* already exists (language - iteration 22). -- **Distributed tracing / OpenTelemetry export.** Needs `net.connect` - (iteration 38) and a wire protocol; a later slice if a consumer appears. +- **CPU/heap profiling** (`pprof`'s equivalent) — its own runtime-v2 iteration + when a consumer measures the need (fork 1). +- **Push / OpenTelemetry export** — needs a collector protocol and a consumer; + `net.connect_tls` now exists, so the transport is no longer the blocker. +- **Cross-shard aggregation** of the gauges — an inbox round-trip; scrape or sum + in the app for now. +- **`expvar`-style JSON**, format flags, per-request latency histograms — later, + on demand. +- **Per-change CI and fuzzing.** Tooling and process, not a runtime surface; + the benchmark *harness* already exists (language iteration 22). - **Alerting, dashboards.** Downstream of exposition, not the runtime's job. ## Info Consumers exist and are named above, so this is not a primitive shipped as -decoration. Ordering: stack-trace-on-trap has no dependency and could lead; -counters/gauges are next; profiling is the heaviest and most separable. Nothing -here depends on the porch track — the dependency runs the other way. +decoration. Ordering: phase A (trace on trap) has no dependency and leads; +phase B (counters) follows; phase C is porch's. Nothing here depends on the +porch track — the dependency runs the other way. Small: a frame walk at two +trap sites, one module builtin reading fields that already exist.