docs(rv2-obs): rv2 7 observability brainstormed to ready

- four forks settled with KISS defaults grounded in runtime/src: counters+gauges only (profiling split out), Prometheus text rendered in .wo from a map<Text, Int>, pull via proc.metrics(), stack trace on trap lands first
- phases A (trace on trap at both trap sites) / B (proc.metrics from existing gc/arena/fiber fields) / C (porch mounts /metrics — consumer's phase)
- builtin id to be confirmed against WO_B_MAX at build time (random_bytes claims 119 per porch 2's brief)
- review_pending marker: forks auto-approved 2026-09-09, developer second review before code lands
- board row: refine -> ready

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit feb11c3aad613ed7f41b280b27c1f6c0dda92ec7)
This commit is contained in:
shoney.arickathil 2026-09-09 16:25:50 +02:00
parent ed45ac7c06
commit e1b9ada190
2 changed files with 92 additions and 42 deletions

View file

@ -1611,7 +1611,7 @@ starts. Edges in [dependency graph section 6](../00-dependency-graph.md).
| 4 | [termios adoption](runtime-v2/04-termios.md) | ✅ **DONE 2026-09-02** — `term.raw/restore`; restore proven a runtime obligation twice (DIV0 while raw, and the double-raw refusal itself) |
| 5 | [fd passing](runtime-v2/05-fd-passing.md) | ✅ **DONE 2026-09-02** — `net.send_fd`/`recv_fd`/`connect_unix`; a tty crossed the socket, was raw'd through the received copy and restored at destroy — the wmux handover in miniature |
| 6 | [term.size + term.width](runtime-v2/06-term-size-width.md) | ✅ **DONE 2026-09-02** — TIOCGWINSZ read twin (nil = not a tty) and libc wcwidth under C.UTF-8; the only runtime work the whole wmux parity ladder needs |
| 7 | [observability](runtime-v2/07-observability.md) | ⬜ `refine` — **moved here 2026-09-06** from language iteration 30 (`was_language_iteration: 30`). Runtime metrics/gauges, a `pprof`-equivalent profile, stack-trace-on-trap; consumers named (porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry). Forks: counters-only vs profiling, exposition format, pull vs push, trace-on-trap as a separable first slice. Stretches the track's charter (instrumentation, not processes/terminals/signals) — noted in the story |
| 7 | [observability](runtime-v2/07-observability.md) | ⬜ **`ready` 2026-09-09** (was `refine`; moved here 2026-09-06 from language iteration 30). Four forks locked, grounded in `runtime/src`: (1) **counters/gauges only** — they already exist as fields (`gc_traced_cnt`, `gc_alloc_bytes`, `gc_step_no`, arena `used`, `nfibers`, `nchildren`, `ntls`); profiling split to its own later iteration; (2) **Prometheus text rendered in `.wo`** from a `map<Text,Int>` (no new record, no JSON); (3) **pull** via `proc.metrics() -> map<Text,Int>`, per shard; push/OTel deferred by name; (4) **stack-trace-on-trap lands first, alone** — the trap path already has method+line (per-method line table) and `vm_unwind` walks frames, so phase A appends `at METHOD line N` per frame under the existing trap line. Phases A trace → B `proc.metrics()` → C porch mounts `/metrics` (porch's). Builtin id confirmed against `WO_B_MAX` at build (shared enum). Consumers: porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry. Forks auto-approved, `review_pending` |
| 8 | [symmetric cipher (AEAD)](runtime-v2/08-symmetric-cipher.md) | 🔄 **in-progress** — the **first rung of the TLS ladder** (gates rv2 9). **Phases A + B + C LANDED 2026-09-08**: A ChaCha20-Poly1305 (ids 111/112, RFC 8439 §2.8.2); B AES-128/256-GCM (ids 113/114) via AES-NI+PCLMULQDQ; C portable constant-time software AES-GCM fallback (S-box via GF-inverse ladder, bit-by-bit GHASH) — AES-GCM now on any CPU, dispatched hw-or-sw. All hand-rolled, constant-time, both AES paths NIST cases 4 & 16 byte-exact, KAT-gated in test_crypto (**48/0**), ASan/UBSan clean. Remaining: D cookie wrapper → E gate (ARMv8 hw path deferred). Consumers: rv2 9 TLS + porch encrypted cookies |
| 9 | [in-process TLS](runtime-v2/09-in-process-tls.md) | ✅ **DONE 2026-09-09** — in-process TLS 1.3 **both directions**, **retired the "TLS is the proxy's job" doctrine** (34/38/porch corrected). Hand-rolled, 1.3-only, RSA+ECDSA+full X.509; KAT'd vs **RFC 8448** / real certs, ASan/UBSan clean. **A–E crypto** (AEAD, HKDF, X25519, sign/verify, X.509 + SAN + basicConstraints/EKU) → **F client** (`net.connect_tls`/`read_tls`/`write_tls`, ids 115–117) → **G server** (constant-time RSA-PSS + ECDSA-P256 signing w/ RFC 6979, server FSM, `net.accept_tls` id 118, `wo_pkey_parse`). Live-gated: `just tls` 5/0 (outbound) + `just tls-server` 4/0 (inbound, openssl s_client EC+RSA). test_tls 123/0, test_crypto 130/0, full suite 0 fail. Deferred follow-ups (non-blocking): park-based handshake, `TlsConn` object, connection pooling, close_notify, complete-formula EC ladder. Forks auto-approved 2026-09-08/09, `review_pending`. The project's **highest-risk** work — done |

View file

@ -3,18 +3,18 @@ track: runtime-v2
iteration: "7"
was_language_iteration: "30"
status: pending
readiness: refine
readiness: ready
review_pending: "forks auto-approved 2026-09-09 for autonomous execution — developer second review before code lands; four forks settled with KISS defaults grounded in runtime/src (existing gauges, existing line table)"
---
# runtime-v2 7 — observability: metrics, profiling, and traces on trap
# runtime-v2 7 — observability: trace on trap, then counters and gauges
> Moved 2026-09-06 from the language track (was language iteration 30, the
> number a dozen docs still point at) into runtime-v2, whose builtin-sized-seam
> shape it fits. It stretches the track's original processes/terminals/signals
> charter — observability is runtime instrumentation of the VM, GC and shards —
> but the track already grew past its first five seams. **`readiness: refine`** —
> the gap, its consumers and its forks are named here, nothing is brainstormed to
> `ready` yet.
> shape it fits. **Brainstormed to `ready` 2026-09-09**: the four forks below are
> settled, grounded in what the runtime already holds rather than in what an
> observability stack usually ships. Profiling is split out (fork 1) — this
> iteration is the cheap, high-value half.
## Why this exists
@ -37,53 +37,103 @@ once:
- The **rate limiter**'s ephemeral-row expiry is lazy "because porch has no
timer and iteration 30 owns" the sweep story (that "iteration 30" is now this
one — [porch 1](../porch/01-store-backed-middleware.md)).
- **Debugging a trap** today is one stderr line — `trap CODE in METHOD at line
N: MESSAGE` — with no call stack, so a trap three calls deep names only the
innermost frame.
Fiber ships `expvar` and `pprof` as middleware; the equivalents here are runtime
work, because the numbers they expose (allocations, fiber counts, shard load,
GC pauses) live in the C runtime, not in `.wo`.
## What it should deliver (scope to be refined)
## Decisions locked (2026-09-09)
- **Runtime counters and gauges** — allocations, arena high-water, live fiber and
actor counts, per-shard load, GC pause totals, request counters — exposed
through one endpoint the app can mount.
- **A profiling story** — CPU and heap sampling, the `pprof` equivalent, so a hot
path can be found rather than guessed at.
- **A stack trace on trap** — today a trap is a 500 and a line; a trace at the
trap site is the cheapest debugging win and may be separable from the metrics
work.
1. **Counters and gauges only; profiling is its own later iteration.** The
gauges this iteration exposes already exist as runtime fields —
`gc_traced_cnt`, `gc_alloc_bytes`, `gc_step_no` (obj.h), the arena's `used`,
`nfibers`, `nchildren`, `ntls` (vm.h) — so exposing them is a read, not new
machinery. CPU/heap profiling needs sampling infrastructure in the VM and is a
different size of commitment; it gets its own runtime-v2 iteration when a
consumer measures the need. (The story always allowed this split.)
2. **Exposition is Prometheus text, rendered in `.wo`.** The runtime returns
numbers; the *format* is the consumer's. The builtin hands back a
`map<Text, Int>` (an existing container — no new record class), and porch
renders the ops-standard Prometheus text (`name value` lines) from it in a
few `.wo` lines. No `expvar`-style JSON in this iteration (a `json.encode` of
the same map is a one-liner if a consumer asks); no format negotiation.
3. **Pull, not push.** A `proc.metrics() -> map<Text, Int>` builtin — `proc`
because it is this process/shard's introspection, beside `proc.run`/`spawn` —
and porch mounts `/metrics` on it. Push to a collector is possible now that
`net.connect`/`net.connect_tls` exist, but it needs a collector protocol and a
consumer; deferred by name. The snapshot is **per shard** (the calling
shard's gauges); a cross-shard aggregate would need an inbox round-trip and
is deferred — scrape each shard or sum in the app.
4. **Stack trace on trap lands first, alone, in this iteration.** Highest
debugging value per line and zero dependency on the metrics half. The trap
path already resolves method + line through the per-method line table
(loader.c validates it ascending); `vm_unwind` already walks the frame stack.
Phase A walks the frames *before* unwinding and prints one ` at METHOD line
N` line per frame under the existing trap line — both the main-fiber trap
(main.c) and the fiber trap (vm.c) sites. Stderr only, always on (a trap is
already a stderr event); no new builtin, no format flag.
## Forks the brainstorm must settle
Builtin id: the next free after the ones being claimed ahead of it
(`random_bytes` takes 119 per porch 2's brief) — **confirm against `WO_B_MAX`
in `runtime/src/wob.h` at build time; the id space is one shared enum**
(wob.h + emit.ml/types.ml + loader.c arity), the lesson porch 2's brief
recorded.
1. **Counters only, or profiling too?** Counters and gauges are a bounded, mostly
`.wo`-plus-a-few-builtins surface; CPU/heap profiling needs sampling
machinery in the runtime and is a much larger commitment. Splitting profiling
into its own iteration is a legitimate outcome.
2. **Exposition format.** Prometheus text (the ops-standard, scrape-friendly),
an `expvar`-style JSON blob, or both. The format decides who can consume it
without a translator.
3. **Pull endpoint or push.** A mounted `/metrics` endpoint (pull) fits the
single-binary model; a push to a collector needs `net.connect` — which now
exists (id 110, landed 2026-09-07; `net.connect_tls` for an HTTPS collector,
runtime-v2 9) — so push is possible, but pull is still the simpler default;
say which.
4. **Is stack-trace-on-trap in this iteration at all?** It is separable, it is
the highest debugging value per line, and it touches the trap path rather than
the metrics path — a candidate to land first and alone.
## Phases
- **A — stack trace on trap.** At both trap-report sites, walk the trapping
fiber's frames innermost-first and print ` at <method> line <n>` per frame
(line from the method's line table at the frame's pc; `?` when a method has
no table). Verify: a corpus/regress fixture that traps three calls deep shows
three `at` lines in order under the trap line; a top-level trap shows one;
every existing fixture's first stderr line is byte-unchanged (the trace is
appended, never prepended).
- **B — `proc.metrics()`.** The builtin fills a `map<Text, Int>` from the
shard's existing fields — at least `arena_used_bytes`, `gc_traced_objects`,
`gc_alloc_bytes_since_cycle`, `gc_slices`, `live_fibers`, `live_children`,
`live_tls_conns`, plus `shard_id` — registered in wob.h + types.ml + loader.c
(module member, `proc`). Verify: a runtime test asserts every key is present
and that `live_fibers` rises with spawned fibers and falls when they finish;
the map's keys are stable names (they are the metric names porch will emit).
- **C — porch mounts `/metrics`** rendering Prometheus text from the map. This is
the **consumer's** phase — pure `.wo`, owned by porch 8's lifecycle slice — and
is named here so the builtin ships with its consumer visible, not built here.
## Acceptance Criteria
- **Given** a program that traps three calls deep, **when** it traps, **then**
stderr carries the existing trap line unchanged followed by three `at METHOD
line N` lines, innermost first.
- **Given** every existing fixture and gate, **when** run, **then** the first
stderr line of each trap is byte-identical to before (the trace only appends).
- **Given** a `.wo` program that spawns N fibers and calls `proc.metrics()`,
**when** inspected, **then** `live_fibers` reflects them and every named key is
present with an `Int`.
- **Given** the map, **when** porch renders it, **then** the output is valid
Prometheus text (one `name value` line per key) — proven in porch's own gate
when phase C lands.
## Out of scope (named, owned elsewhere)
- **Per-change CI and fuzzing.** Frequently lumped under the old "iteration 30"
but they are tooling and process, not a runtime surface; they belong to a
CI/ops story, not this one. The benchmark *harness* already exists (language
iteration 22).
- **Distributed tracing / OpenTelemetry export.** Needs `net.connect`
(iteration 38) and a wire protocol; a later slice if a consumer appears.
- **CPU/heap profiling** (`pprof`'s equivalent) — its own runtime-v2 iteration
when a consumer measures the need (fork 1).
- **Push / OpenTelemetry export** — needs a collector protocol and a consumer;
`net.connect_tls` now exists, so the transport is no longer the blocker.
- **Cross-shard aggregation** of the gauges — an inbox round-trip; scrape or sum
in the app for now.
- **`expvar`-style JSON**, format flags, per-request latency histograms — later,
on demand.
- **Per-change CI and fuzzing.** Tooling and process, not a runtime surface;
the benchmark *harness* already exists (language iteration 22).
- **Alerting, dashboards.** Downstream of exposition, not the runtime's job.
## Info
Consumers exist and are named above, so this is not a primitive shipped as
decoration. Ordering: stack-trace-on-trap has no dependency and could lead;
counters/gauges are next; profiling is the heaviest and most separable. Nothing
here depends on the porch track — the dependency runs the other way.
decoration. Ordering: phase A (trace on trap) has no dependency and leads;
phase B (counters) follows; phase C is porch's. Nothing here depends on the
porch track — the dependency runs the other way. Small: a frame walk at two
trap sites, one module builtin reading fields that already exist.