docs(rv2-obs): rv2 7 observability brainstormed to ready
- four forks settled with KISS defaults grounded in runtime/src: counters+gauges only (profiling split out), Prometheus text rendered in .wo from a map<Text, Int>, pull via proc.metrics(), stack trace on trap lands first - phases A (trace on trap at both trap sites) / B (proc.metrics from existing gc/arena/fiber fields) / C (porch mounts /metrics — consumer's phase) - builtin id to be confirmed against WO_B_MAX at build time (random_bytes claims 119 per porch 2's brief) - review_pending marker: forks auto-approved 2026-09-09, developer second review before code lands - board row: refine -> ready Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit feb11c3aad613ed7f41b280b27c1f6c0dda92ec7)
This commit is contained in:
parent
ed45ac7c06
commit
e1b9ada190
2 changed files with 92 additions and 42 deletions
|
|
@ -1611,7 +1611,7 @@ starts. Edges in [dependency graph section 6](../00-dependency-graph.md).
|
||||||
| 4 | [termios adoption](runtime-v2/04-termios.md) | ✅ **DONE 2026-09-02** — `term.raw/restore`; restore proven a runtime obligation twice (DIV0 while raw, and the double-raw refusal itself) |
|
| 4 | [termios adoption](runtime-v2/04-termios.md) | ✅ **DONE 2026-09-02** — `term.raw/restore`; restore proven a runtime obligation twice (DIV0 while raw, and the double-raw refusal itself) |
|
||||||
| 5 | [fd passing](runtime-v2/05-fd-passing.md) | ✅ **DONE 2026-09-02** — `net.send_fd`/`recv_fd`/`connect_unix`; a tty crossed the socket, was raw'd through the received copy and restored at destroy — the wmux handover in miniature |
|
| 5 | [fd passing](runtime-v2/05-fd-passing.md) | ✅ **DONE 2026-09-02** — `net.send_fd`/`recv_fd`/`connect_unix`; a tty crossed the socket, was raw'd through the received copy and restored at destroy — the wmux handover in miniature |
|
||||||
| 6 | [term.size + term.width](runtime-v2/06-term-size-width.md) | ✅ **DONE 2026-09-02** — TIOCGWINSZ read twin (nil = not a tty) and libc wcwidth under C.UTF-8; the only runtime work the whole wmux parity ladder needs |
|
| 6 | [term.size + term.width](runtime-v2/06-term-size-width.md) | ✅ **DONE 2026-09-02** — TIOCGWINSZ read twin (nil = not a tty) and libc wcwidth under C.UTF-8; the only runtime work the whole wmux parity ladder needs |
|
||||||
| 7 | [observability](runtime-v2/07-observability.md) | ⬜ `refine` — **moved here 2026-09-06** from language iteration 30 (`was_language_iteration: 30`). Runtime metrics/gauges, a `pprof`-equivalent profile, stack-trace-on-trap; consumers named (porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry). Forks: counters-only vs profiling, exposition format, pull vs push, trace-on-trap as a separable first slice. Stretches the track's charter (instrumentation, not processes/terminals/signals) — noted in the story |
|
| 7 | [observability](runtime-v2/07-observability.md) | ⬜ **`ready` 2026-09-09** (was `refine`; moved here 2026-09-06 from language iteration 30). Four forks locked, grounded in `runtime/src`: (1) **counters/gauges only** — they already exist as fields (`gc_traced_cnt`, `gc_alloc_bytes`, `gc_step_no`, arena `used`, `nfibers`, `nchildren`, `ntls`); profiling split to its own later iteration; (2) **Prometheus text rendered in `.wo`** from a `map<Text,Int>` (no new record, no JSON); (3) **pull** via `proc.metrics() -> map<Text,Int>`, per shard; push/OTel deferred by name; (4) **stack-trace-on-trap lands first, alone** — the trap path already has method+line (per-method line table) and `vm_unwind` walks frames, so phase A appends `at METHOD line N` per frame under the existing trap line. Phases A trace → B `proc.metrics()` → C porch mounts `/metrics` (porch's). Builtin id confirmed against `WO_B_MAX` at build (shared enum). Consumers: porch [8](porch/08-static-and-lifecycle.md)/[39](language-runtime-database/39-web-framework-parity.md), databasev2 [5](databasev2/05-bounded-tables-eviction.md), the limiter's lazy expiry. Forks auto-approved, `review_pending` |
|
||||||
| 8 | [symmetric cipher (AEAD)](runtime-v2/08-symmetric-cipher.md) | 🔄 **in-progress** — the **first rung of the TLS ladder** (gates rv2 9). **Phases A + B + C LANDED 2026-09-08**: A ChaCha20-Poly1305 (ids 111/112, RFC 8439 §2.8.2); B AES-128/256-GCM (ids 113/114) via AES-NI+PCLMULQDQ; C portable constant-time software AES-GCM fallback (S-box via GF-inverse ladder, bit-by-bit GHASH) — AES-GCM now on any CPU, dispatched hw-or-sw. All hand-rolled, constant-time, both AES paths NIST cases 4 & 16 byte-exact, KAT-gated in test_crypto (**48/0**), ASan/UBSan clean. Remaining: D cookie wrapper → E gate (ARMv8 hw path deferred). Consumers: rv2 9 TLS + porch encrypted cookies |
|
| 8 | [symmetric cipher (AEAD)](runtime-v2/08-symmetric-cipher.md) | 🔄 **in-progress** — the **first rung of the TLS ladder** (gates rv2 9). **Phases A + B + C LANDED 2026-09-08**: A ChaCha20-Poly1305 (ids 111/112, RFC 8439 §2.8.2); B AES-128/256-GCM (ids 113/114) via AES-NI+PCLMULQDQ; C portable constant-time software AES-GCM fallback (S-box via GF-inverse ladder, bit-by-bit GHASH) — AES-GCM now on any CPU, dispatched hw-or-sw. All hand-rolled, constant-time, both AES paths NIST cases 4 & 16 byte-exact, KAT-gated in test_crypto (**48/0**), ASan/UBSan clean. Remaining: D cookie wrapper → E gate (ARMv8 hw path deferred). Consumers: rv2 9 TLS + porch encrypted cookies |
|
||||||
| 9 | [in-process TLS](runtime-v2/09-in-process-tls.md) | ✅ **DONE 2026-09-09** — in-process TLS 1.3 **both directions**, **retired the "TLS is the proxy's job" doctrine** (34/38/porch corrected). Hand-rolled, 1.3-only, RSA+ECDSA+full X.509; KAT'd vs **RFC 8448** / real certs, ASan/UBSan clean. **A–E crypto** (AEAD, HKDF, X25519, sign/verify, X.509 + SAN + basicConstraints/EKU) → **F client** (`net.connect_tls`/`read_tls`/`write_tls`, ids 115–117) → **G server** (constant-time RSA-PSS + ECDSA-P256 signing w/ RFC 6979, server FSM, `net.accept_tls` id 118, `wo_pkey_parse`). Live-gated: `just tls` 5/0 (outbound) + `just tls-server` 4/0 (inbound, openssl s_client EC+RSA). test_tls 123/0, test_crypto 130/0, full suite 0 fail. Deferred follow-ups (non-blocking): park-based handshake, `TlsConn` object, connection pooling, close_notify, complete-formula EC ladder. Forks auto-approved 2026-09-08/09, `review_pending`. The project's **highest-risk** work — done |
|
| 9 | [in-process TLS](runtime-v2/09-in-process-tls.md) | ✅ **DONE 2026-09-09** — in-process TLS 1.3 **both directions**, **retired the "TLS is the proxy's job" doctrine** (34/38/porch corrected). Hand-rolled, 1.3-only, RSA+ECDSA+full X.509; KAT'd vs **RFC 8448** / real certs, ASan/UBSan clean. **A–E crypto** (AEAD, HKDF, X25519, sign/verify, X.509 + SAN + basicConstraints/EKU) → **F client** (`net.connect_tls`/`read_tls`/`write_tls`, ids 115–117) → **G server** (constant-time RSA-PSS + ECDSA-P256 signing w/ RFC 6979, server FSM, `net.accept_tls` id 118, `wo_pkey_parse`). Live-gated: `just tls` 5/0 (outbound) + `just tls-server` 4/0 (inbound, openssl s_client EC+RSA). test_tls 123/0, test_crypto 130/0, full suite 0 fail. Deferred follow-ups (non-blocking): park-based handshake, `TlsConn` object, connection pooling, close_notify, complete-formula EC ladder. Forks auto-approved 2026-09-08/09, `review_pending`. The project's **highest-risk** work — done |
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,18 +3,18 @@ track: runtime-v2
|
||||||
iteration: "7"
|
iteration: "7"
|
||||||
was_language_iteration: "30"
|
was_language_iteration: "30"
|
||||||
status: pending
|
status: pending
|
||||||
readiness: refine
|
readiness: ready
|
||||||
|
review_pending: "forks auto-approved 2026-09-09 for autonomous execution — developer second review before code lands; four forks settled with KISS defaults grounded in runtime/src (existing gauges, existing line table)"
|
||||||
---
|
---
|
||||||
|
|
||||||
# runtime-v2 7 — observability: metrics, profiling, and traces on trap
|
# runtime-v2 7 — observability: trace on trap, then counters and gauges
|
||||||
|
|
||||||
> Moved 2026-09-06 from the language track (was language iteration 30, the
|
> Moved 2026-09-06 from the language track (was language iteration 30, the
|
||||||
> number a dozen docs still point at) into runtime-v2, whose builtin-sized-seam
|
> number a dozen docs still point at) into runtime-v2, whose builtin-sized-seam
|
||||||
> shape it fits. It stretches the track's original processes/terminals/signals
|
> shape it fits. **Brainstormed to `ready` 2026-09-09**: the four forks below are
|
||||||
> charter — observability is runtime instrumentation of the VM, GC and shards —
|
> settled, grounded in what the runtime already holds rather than in what an
|
||||||
> but the track already grew past its first five seams. **`readiness: refine`** —
|
> observability stack usually ships. Profiling is split out (fork 1) — this
|
||||||
> the gap, its consumers and its forks are named here, nothing is brainstormed to
|
> iteration is the cheap, high-value half.
|
||||||
> `ready` yet.
|
|
||||||
|
|
||||||
## Why this exists
|
## Why this exists
|
||||||
|
|
||||||
|
|
@ -37,53 +37,103 @@ once:
|
||||||
- The **rate limiter**'s ephemeral-row expiry is lazy "because porch has no
|
- The **rate limiter**'s ephemeral-row expiry is lazy "because porch has no
|
||||||
timer and iteration 30 owns" the sweep story (that "iteration 30" is now this
|
timer and iteration 30 owns" the sweep story (that "iteration 30" is now this
|
||||||
one — [porch 1](../porch/01-store-backed-middleware.md)).
|
one — [porch 1](../porch/01-store-backed-middleware.md)).
|
||||||
|
- **Debugging a trap** today is one stderr line — `trap CODE in METHOD at line
|
||||||
|
N: MESSAGE` — with no call stack, so a trap three calls deep names only the
|
||||||
|
innermost frame.
|
||||||
|
|
||||||
Fiber ships `expvar` and `pprof` as middleware; the equivalents here are runtime
|
Fiber ships `expvar` and `pprof` as middleware; the equivalents here are runtime
|
||||||
work, because the numbers they expose (allocations, fiber counts, shard load,
|
work, because the numbers they expose (allocations, fiber counts, shard load,
|
||||||
GC pauses) live in the C runtime, not in `.wo`.
|
GC pauses) live in the C runtime, not in `.wo`.
|
||||||
|
|
||||||
## What it should deliver (scope to be refined)
|
## Decisions locked (2026-09-09)
|
||||||
|
|
||||||
- **Runtime counters and gauges** — allocations, arena high-water, live fiber and
|
1. **Counters and gauges only; profiling is its own later iteration.** The
|
||||||
actor counts, per-shard load, GC pause totals, request counters — exposed
|
gauges this iteration exposes already exist as runtime fields —
|
||||||
through one endpoint the app can mount.
|
`gc_traced_cnt`, `gc_alloc_bytes`, `gc_step_no` (obj.h), the arena's `used`,
|
||||||
- **A profiling story** — CPU and heap sampling, the `pprof` equivalent, so a hot
|
`nfibers`, `nchildren`, `ntls` (vm.h) — so exposing them is a read, not new
|
||||||
path can be found rather than guessed at.
|
machinery. CPU/heap profiling needs sampling infrastructure in the VM and is a
|
||||||
- **A stack trace on trap** — today a trap is a 500 and a line; a trace at the
|
different size of commitment; it gets its own runtime-v2 iteration when a
|
||||||
trap site is the cheapest debugging win and may be separable from the metrics
|
consumer measures the need. (The story always allowed this split.)
|
||||||
work.
|
2. **Exposition is Prometheus text, rendered in `.wo`.** The runtime returns
|
||||||
|
numbers; the *format* is the consumer's. The builtin hands back a
|
||||||
|
`map<Text, Int>` (an existing container — no new record class), and porch
|
||||||
|
renders the ops-standard Prometheus text (`name value` lines) from it in a
|
||||||
|
few `.wo` lines. No `expvar`-style JSON in this iteration (a `json.encode` of
|
||||||
|
the same map is a one-liner if a consumer asks); no format negotiation.
|
||||||
|
3. **Pull, not push.** A `proc.metrics() -> map<Text, Int>` builtin — `proc`
|
||||||
|
because it is this process/shard's introspection, beside `proc.run`/`spawn` —
|
||||||
|
and porch mounts `/metrics` on it. Push to a collector is possible now that
|
||||||
|
`net.connect`/`net.connect_tls` exist, but it needs a collector protocol and a
|
||||||
|
consumer; deferred by name. The snapshot is **per shard** (the calling
|
||||||
|
shard's gauges); a cross-shard aggregate would need an inbox round-trip and
|
||||||
|
is deferred — scrape each shard or sum in the app.
|
||||||
|
4. **Stack trace on trap lands first, alone, in this iteration.** Highest
|
||||||
|
debugging value per line and zero dependency on the metrics half. The trap
|
||||||
|
path already resolves method + line through the per-method line table
|
||||||
|
(loader.c validates it ascending); `vm_unwind` already walks the frame stack.
|
||||||
|
Phase A walks the frames *before* unwinding and prints one ` at METHOD line
|
||||||
|
N` line per frame under the existing trap line — both the main-fiber trap
|
||||||
|
(main.c) and the fiber trap (vm.c) sites. Stderr only, always on (a trap is
|
||||||
|
already a stderr event); no new builtin, no format flag.
|
||||||
|
|
||||||
## Forks the brainstorm must settle
|
Builtin id: the next free after the ones being claimed ahead of it
|
||||||
|
(`random_bytes` takes 119 per porch 2's brief) — **confirm against `WO_B_MAX`
|
||||||
|
in `runtime/src/wob.h` at build time; the id space is one shared enum**
|
||||||
|
(wob.h + emit.ml/types.ml + loader.c arity), the lesson porch 2's brief
|
||||||
|
recorded.
|
||||||
|
|
||||||
1. **Counters only, or profiling too?** Counters and gauges are a bounded, mostly
|
## Phases
|
||||||
`.wo`-plus-a-few-builtins surface; CPU/heap profiling needs sampling
|
|
||||||
machinery in the runtime and is a much larger commitment. Splitting profiling
|
- **A — stack trace on trap.** At both trap-report sites, walk the trapping
|
||||||
into its own iteration is a legitimate outcome.
|
fiber's frames innermost-first and print ` at <method> line <n>` per frame
|
||||||
2. **Exposition format.** Prometheus text (the ops-standard, scrape-friendly),
|
(line from the method's line table at the frame's pc; `?` when a method has
|
||||||
an `expvar`-style JSON blob, or both. The format decides who can consume it
|
no table). Verify: a corpus/regress fixture that traps three calls deep shows
|
||||||
without a translator.
|
three `at` lines in order under the trap line; a top-level trap shows one;
|
||||||
3. **Pull endpoint or push.** A mounted `/metrics` endpoint (pull) fits the
|
every existing fixture's first stderr line is byte-unchanged (the trace is
|
||||||
single-binary model; a push to a collector needs `net.connect` — which now
|
appended, never prepended).
|
||||||
exists (id 110, landed 2026-09-07; `net.connect_tls` for an HTTPS collector,
|
- **B — `proc.metrics()`.** The builtin fills a `map<Text, Int>` from the
|
||||||
runtime-v2 9) — so push is possible, but pull is still the simpler default;
|
shard's existing fields — at least `arena_used_bytes`, `gc_traced_objects`,
|
||||||
say which.
|
`gc_alloc_bytes_since_cycle`, `gc_slices`, `live_fibers`, `live_children`,
|
||||||
4. **Is stack-trace-on-trap in this iteration at all?** It is separable, it is
|
`live_tls_conns`, plus `shard_id` — registered in wob.h + types.ml + loader.c
|
||||||
the highest debugging value per line, and it touches the trap path rather than
|
(module member, `proc`). Verify: a runtime test asserts every key is present
|
||||||
the metrics path — a candidate to land first and alone.
|
and that `live_fibers` rises with spawned fibers and falls when they finish;
|
||||||
|
the map's keys are stable names (they are the metric names porch will emit).
|
||||||
|
- **C — porch mounts `/metrics`** rendering Prometheus text from the map. This is
|
||||||
|
the **consumer's** phase — pure `.wo`, owned by porch 8's lifecycle slice — and
|
||||||
|
is named here so the builtin ships with its consumer visible, not built here.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- **Given** a program that traps three calls deep, **when** it traps, **then**
|
||||||
|
stderr carries the existing trap line unchanged followed by three `at METHOD
|
||||||
|
line N` lines, innermost first.
|
||||||
|
- **Given** every existing fixture and gate, **when** run, **then** the first
|
||||||
|
stderr line of each trap is byte-identical to before (the trace only appends).
|
||||||
|
- **Given** a `.wo` program that spawns N fibers and calls `proc.metrics()`,
|
||||||
|
**when** inspected, **then** `live_fibers` reflects them and every named key is
|
||||||
|
present with an `Int`.
|
||||||
|
- **Given** the map, **when** porch renders it, **then** the output is valid
|
||||||
|
Prometheus text (one `name value` line per key) — proven in porch's own gate
|
||||||
|
when phase C lands.
|
||||||
|
|
||||||
## Out of scope (named, owned elsewhere)
|
## Out of scope (named, owned elsewhere)
|
||||||
|
|
||||||
- **Per-change CI and fuzzing.** Frequently lumped under the old "iteration 30"
|
- **CPU/heap profiling** (`pprof`'s equivalent) — its own runtime-v2 iteration
|
||||||
but they are tooling and process, not a runtime surface; they belong to a
|
when a consumer measures the need (fork 1).
|
||||||
CI/ops story, not this one. The benchmark *harness* already exists (language
|
- **Push / OpenTelemetry export** — needs a collector protocol and a consumer;
|
||||||
iteration 22).
|
`net.connect_tls` now exists, so the transport is no longer the blocker.
|
||||||
- **Distributed tracing / OpenTelemetry export.** Needs `net.connect`
|
- **Cross-shard aggregation** of the gauges — an inbox round-trip; scrape or sum
|
||||||
(iteration 38) and a wire protocol; a later slice if a consumer appears.
|
in the app for now.
|
||||||
|
- **`expvar`-style JSON**, format flags, per-request latency histograms — later,
|
||||||
|
on demand.
|
||||||
|
- **Per-change CI and fuzzing.** Tooling and process, not a runtime surface;
|
||||||
|
the benchmark *harness* already exists (language iteration 22).
|
||||||
- **Alerting, dashboards.** Downstream of exposition, not the runtime's job.
|
- **Alerting, dashboards.** Downstream of exposition, not the runtime's job.
|
||||||
|
|
||||||
## Info
|
## Info
|
||||||
|
|
||||||
Consumers exist and are named above, so this is not a primitive shipped as
|
Consumers exist and are named above, so this is not a primitive shipped as
|
||||||
decoration. Ordering: stack-trace-on-trap has no dependency and could lead;
|
decoration. Ordering: phase A (trace on trap) has no dependency and leads;
|
||||||
counters/gauges are next; profiling is the heaviest and most separable. Nothing
|
phase B (counters) follows; phase C is porch's. Nothing here depends on the
|
||||||
here depends on the porch track — the dependency runs the other way.
|
porch track — the dependency runs the other way. Small: a frame walk at two
|
||||||
|
trap sites, one module builtin reading fields that already exist.
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue