shoney/writeonce
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix(vm): marshal cross-shard actor messages (language 41) — the double free

Browse source 
Root cause (decision 1): cross-shard send/call/monitor pointer-shared the
message into the receiver's shard (e->payload = msg_val), so a worker read and
eventually dropped an object living in the sender's arena — a double free, then
a class-0 forge, then a modulo self-route livelock, all downstream of that one
broken invariant ("VM heaps are never read cross-shard", which wo_db_rpc keeps).

- actor_marshal: the sender encodes the message into an arena-independent neutral
  form (wo_db_val_encode, the same marshal wo_db_rpc uses) and drops its own
  original — no pointer crosses an arena boundary, so the double-free class is
  gone by construction. actor_unmarshal rebuilds it in the receiver's arena
  (wo_val_decode_vm) and frees the neutral. Applied to the 4 cross-shard
  producers (send x2, call, monitor) + the 3 consumers (kinds 0/5/7). Same-shard
  paths untouched (the WO_SHARDS=1 fast path never failed). Call replies are
  scalars by contract, so kind 6 needs no marshal.
- eng_settle_inboxes: undrained kind-0/5/7 payloads at teardown are the neutral
  form now — free with wo_db_val_free, not wo_drop_obj (caught by ASan mid-fix).
- decision 2: wo_route_free traps a shard_id >= nshards header (a corrupt/freed
  block) instead of self-routing it into the settle livelock.
- proof: tests/regress/lang-41/cross-shard-marshal.wo (a multi<Text> sent +
  called cross-shard, both sides drop) — clean 12x/5x under WO_SHARDS=4 + ASan;
  shard-settle repro still clean 8x; full runtime suite 0 fail (same-shard
  byte-unchanged). `just db-actor` extended with the new fixture.
- unblocks porch 9. Follow-ups: poison-on-free (decision 3), corpus fixture (4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 63065ff75799f7f43b2bce6de61e77856799566f)
This commit is contained in:
shoney.arickathil 2026-09-09 10:14:37 +02:00
parent 8992dbd589
commit 00214bd68e
5 changed files with 190 additions and 16 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
docs/stories/00-status.md
View file

@ -76,6 +76,20 @@ behind this board; live Obsidian Dataview views:
## ▶ NEXT PLAN
### Landed 2026-09-09 — language 41 fixed (cross-shard message marshal); porch 9 unblocked
The actor-arena double-free/hang is fixed. Cross-shard `send`/`call`/monitor now
**marshal** the message (encode to a neutral form on the sender via
`wo_db_val_encode`, rebuild in the receiver's arena via `wo_val_decode_vm`) —
no pointer crosses an arena boundary, so the double free is gone by construction
(decision 1). `wo_route_free` now traps a `shard_id >= nshards` header instead of
self-routing it into the settle livelock (decision 2). Proven: new fixture
`tests/regress/lang-41/cross-shard-marshal.wo` (a `multi<Text>` sent + called
cross-shard, both drop) clean 12×/5× under `WO_SHARDS=4` + ASan, the shard-settle
repro still clean, full suite 0 fail (`just db-actor` gate extended). Follow-ups:
poison-on-free (decision 3), a corpus fixture (decision 4). **[porch 9](porch/09-idempotent-replay.md)
is unblocked** — the porch track (2–8 `ready`) is now clear to build.
### Landed 2026-09-09 — rv2 9 in-process TLS 1.3 is COMPLETE, both directions, live-gated
**rv2 9 DONE.** In-process TLS 1.3 both directions, hand-rolled, RFC-8448/real-cert

15
docs/stories/language-runtime-database/41-actor-arena-crash.md
View file

@ -1,10 +1,23 @@
---
track: language-runtime-database
iteration: "41"
status: in-progress
status: done
readiness: ready
---
> ✅ **LANDED 2026-09-09.** The marshal fix (decision 1) + the modulo/bounds
> guard (decision 2) shipped. Cross-shard `send`/`call`/monitor now copy the
> message into a neutral form on the sender (`wo_db_val_encode`) and the receiver
> rebuilds it in its own arena (`wo_val_decode_vm`) — no pointer crosses an arena
> boundary, so the double free is gone by construction; `wo_route_free` traps a
> `shard_id >= nshards` header instead of self-routing it into the settle
> livelock. Proven with a new fixture `tests/regress/lang-41/cross-shard-marshal.wo`
> (a `multi<Text>` sent + called cross-shard, both sides drop) — 12×/5× clean
> under `WO_SHARDS=4` + ASan, the shard-settle repro still clean, full runtime
> suite 0 fail (same-shard paths unchanged). Unblocks
> [porch 9](../porch/09-idempotent-replay.md). Follow-ups (decisions 3/4):
> poison-on-free and a deterministic corpus fixture.
# 41 — the actor arena crash: a SIGSEGV under concurrent parked callers
> Found 2026-08-30 while implementing [porch 1](../porch/01-store-backed-middleware.md).

105
runtime/src/vm.c
View file

@ -82,6 +82,8 @@ static void actor_drop_payload(wo_vm *vm, uint64_t payload);
static void monitors_fire(wo_vm *vm, wo_actor *a);
static void runtime_notify(wo_vm *vm, wo_actor *target, uint64_t msg_val,
const char *what);
static uint64_t actor_marshal(wo_vm *vm, uint64_t msg_val, int *ok); /* lang-41 */
static uint64_t actor_unmarshal(wo_vm *vm, uint64_t neutral); /* lang-41 */
/* the owning thread drains its inbox: adopt actors, deliver sends,
* execute home-routed frees. Returns how many envelopes were handled. */
@ -109,16 +111,17 @@ static int wo_vm_adopt(wo_vm *vm) {
push (OOM) must hand it back or the slot leaks forever.
A dead target drops the moved message silently (the
send-to-dead rule) and frees the slot. */
uint64_t obj0 = actor_unmarshal(vm, e->payload); /* rebuild in our arena */
if (e->actor->dead) {
actor_drop_payload(vm, e->payload);
actor_drop_payload(vm, obj0);
wo_mbox_release(e->actor);
break;
}
wo_msg m0 = { e->payload, NULL, 0 };
wo_msg m0 = { obj0, NULL, 0 };
if (actor_push(e->actor, m0) == 0) {
if (!e->actor->active) (void)actor_activate(vm, e->actor);
} else {
actor_drop_payload(vm, e->payload);
actor_drop_payload(vm, obj0);
wo_mbox_release(e->actor);
}
break;
@ -126,17 +129,18 @@ static int wo_vm_adopt(wo_vm *vm) {
case 5: { /* iteration 24: a cross-shard call — same enqueue as a
send, but the slot remembers the parked caller. A dead
target answers the error reply instead. */
uint64_t objc = actor_unmarshal(vm, e->payload); /* rebuild in our arena */
if (e->actor->dead) {
actor_drop_payload(vm, e->payload);
actor_drop_payload(vm, objc);
wo_mbox_release(e->actor);
call_reply_to(vm, e->from_fiber, e->from_shard, 0, WO_T_ACTOR);
break;
}
wo_msg mc = { e->payload, e->from_fiber, e->from_shard };
wo_msg mc = { objc, e->from_fiber, e->from_shard };
if (actor_push(e->actor, mc) == 0) {
if (!e->actor->active) (void)actor_activate(vm, e->actor);
} else {
actor_drop_payload(vm, e->payload);
actor_drop_payload(vm, objc);
wo_mbox_release(e->actor);
call_reply_to(vm, e->from_fiber, e->from_shard, 0, WO_T_ACTOR);
}
@ -146,17 +150,18 @@ static int wo_vm_adopt(wo_vm *vm) {
WE are the watched actor's home. Dead already = the
notice fires now; else it joins the list. */
wo_actor *ob = (wo_actor *)(uintptr_t)e->from_fiber;
uint64_t objm = actor_unmarshal(vm, e->payload); /* rebuild in our arena */
if (e->actor->dead) {
runtime_notify(vm, ob, e->payload, "death notice");
runtime_notify(vm, ob, objm, "death notice");
break;
}
wo_monitor *mn = calloc(1, sizeof *mn);
if (!mn) {
actor_drop_payload(vm, e->payload);
actor_drop_payload(vm, objm);
break;
}
mn->observer = ob;
mn->msg = e->payload;
mn->msg = objm;
mn->next = e->actor->monitors;
e->actor->monitors = mn;
break;
@ -268,6 +273,19 @@ static int wo_vm_adopt(wo_vm *vm) {
* the header's shard id is not the current thread's) */
void wo_route_free(wo_hdr *h) {
if (eng_teardown) return; /* arenas are torn down wholesale */
/* lang-41 decision 2: a live object's shard_id is always < nshards. An
* out-of-range id is a corrupt or already-freed header (a freelist link
* forged into the header's first bytes) — and inbox_push_to would mask it by
* WO_ENG_MAX_SHARDS into a live inbox that judges it foreign and routes it
* forever (the settle livelock). Trap loudly instead of self-routing. With
* the marshal fix (decision 1) this is unreachable; it is the guard that
* turns any future header corruption into an immediate diagnostic. */
if (h->shard_id >= wo_eng.nshards) {
fprintf(stderr, "wovm: FATAL corrupt or freed object on the free path "
"(shard_id=%u >= nshards=%u, class_id=%u) — not routing\n",
(unsigned)h->shard_id, wo_eng.nshards, (unsigned)h->class_id);
abort();
}
wo_envelope *e = calloc(1, sizeof *e);
if (!e) return; /* OOM on the free path: leak rather than crash */
e->kind = 2;
@ -619,9 +637,11 @@ static int eng_settle_inboxes(void) {
break;
case 0:
case 5:
case 7: /* in-flight payloads: drop (may route -> next pass) */
case 7: /* lang-41: in-flight send/call/monitor payloads are the
MARSHALED neutral form (an arena-independent db_rec), not
a VM object — free them as such, never via wo_drop_obj. */
if (e->payload)
wo_drop_obj(&vm->rt, (wo_hdr *)(uintptr_t)e->payload);
wo_db_val_free(NULL, WO_K_OWNED, e->payload);
break;
case 1: /* an unadopted actor shell */
if (e->actor) {
@ -961,6 +981,33 @@ static void actor_drop_payload(wo_vm *vm, uint64_t payload) {
if (payload) wo_drop_obj(&vm->rt, (wo_hdr *)(uintptr_t)payload);
}
/* lang-41: marshal a cross-shard message. VM heaps are never read cross-shard
* (the wo_db_rpc invariant), so a cross-shard send/call/monitor encodes the
* message into an arena-independent neutral form on the SENDER (a deep copy,
* mirroring wo_db_rpc's arg marshaling) and drops its own original — the pointer
* never crosses an arena boundary, so the double-free class is gone by
* construction. Returns the neutral payload; *ok=0 on OOM (original NOT dropped).
* A nil message (0) marshals to 0. */
static uint64_t actor_marshal(wo_vm *vm, uint64_t msg_val, int *ok) {
*ok = 1;
if (!msg_val) return 0;
const char *m = NULL;
uint64_t neutral = wo_db_val_encode(vm->mod->classes, WO_K_OWNED, msg_val, ok, &m);
if (*ok) wo_drop_obj(&vm->rt, (wo_hdr *)(uintptr_t)msg_val); /* move: drop original */
return neutral;
}
/* lang-41: the receiver half — rebuild a marshaled message in THIS shard's
* arena and free the neutral form. Returns the VM object (0 on nil or OOM). */
static uint64_t actor_unmarshal(wo_vm *vm, uint64_t neutral) {
if (!neutral) return 0;
int ok = 1;
const char *m = NULL;
uint64_t obj = wo_val_decode_vm(NULL, &vm->rt, WO_K_OWNED, neutral, &ok, &m);
wo_db_val_free(NULL, WO_K_OWNED, neutral);
return ok ? obj : 0;
}
/* iteration 24: answer one parked caller. Same-shard callers unpark
* directly; remote ones get a kind-6 envelope. status 0 delivers the
* scalar reply; WO_T_ACTOR makes the caller's re-executed builtin trap. */
@ -1105,9 +1152,16 @@ int wo_vm_actor_send(wo_vm *vm, uint64_t addr, uint64_t msg_val, const char **ms
*msg = "out of memory";
return WO_T_OOM;
}
int mok;
e->payload = actor_marshal(vm, msg_val, &mok); /* copy into a neutral form */
if (!mok) {
free(e);
wo_mbox_release(a);
*msg = "out of memory";
return WO_T_OOM;
}
e->kind = 0;
e->actor = a;
e->payload = msg_val;
inbox_push_to(a->home, e);
return 0;
}
@ -1147,9 +1201,16 @@ static void runtime_notify(wo_vm *vm, wo_actor *target, uint64_t msg_val,
actor_drop_payload(vm, msg_val);
return;
}
int mok;
e->payload = actor_marshal(vm, msg_val, &mok);
if (!mok) {
free(e);
wo_mbox_release(target);
actor_drop_payload(vm, msg_val); /* marshal failed: original still ours */
return;
}
e->kind = 0;
e->actor = target;
e->payload = msg_val;
inbox_push_to(target->home, e);
return;
}
@ -1220,9 +1281,16 @@ int wo_vm_actor_call(wo_vm *vm, uint64_t *R, uint32_t ins, const char **msg) {
*msg = "out of memory";
return WO_T_OOM;
}
int mok;
e->payload = actor_marshal(vm, msg_val, &mok);
if (!mok) {
free(e);
wo_mbox_release(a);
*msg = "out of memory";
return WO_T_OOM;
}
e->kind = 5;
e->actor = a;
e->payload = msg_val;
e->from_shard = vm->shard_id;
e->from_fiber = fb;
inbox_push_to(a->home, e);
@ -1264,9 +1332,16 @@ int wo_vm_actor_monitor(wo_vm *vm, uint64_t watched, uint64_t observer,
*msg = "out of memory";
return WO_T_OOM;
}
int mok;
e->payload = actor_marshal(vm, msg_val, &mok);
if (!mok) {
free(e);
actor_drop_payload(vm, msg_val); /* marshal failed: original still ours */
*msg = "out of memory";
return WO_T_OOM;
}
e->kind = 7;
e->actor = w;
e->payload = msg_val;
e->from_fiber = (wo_fiber *)o; /* reused slot: the observer */
inbox_push_to(w->home, e);
return 0;

24
scripts/db-actor-accept.sh
View file

@ -101,6 +101,30 @@ else
bad "lang-41 shard settle" "fixture did not compile"
fi
# lang-41 phase C: a cross-shard message carrying an OWNED SUBTREE (multi<Text>)
# sent + called into an actor on another shard. Pre-marshal-fix this pointer-
# shared the subtree across arenas -> a double free (ASan abort or settle hang);
# the marshal fix copies it per crossing. Run repeatedly — the failure was
# intermittent (~1 in 6).
L41_CSM="$ROOT/tests/regress/lang-41/cross-shard-marshal.wo"
if [[ -x "$L41_VM" ]] && "$L41_WOC" --emit "$L41_CSM" -o "$L41_W/csm.wob" >/dev/null 2>&1; then
mkdir -p "$L41_W/csmdata"
csm_bad=""
for _ in 1 2 3 4 5; do
csm_out="$(WO_DATA="$L41_W/csmdata" WO_SHARDS=4 timeout 60 "$L41_VM" "$L41_W/csm.wob" 2>&1)"
if grep -q 'SEGV\|AddressSanitizer\|FATAL' <<<"$csm_out" || ! grep -q 'dispatched' <<<"$csm_out"; then
csm_bad="$(grep -m1 'ERROR\|FATAL' <<<"$csm_out"); ${csm_bad}"
fi
done
if [[ -z "$csm_bad" ]]; then
ok "lang-41: cross-shard owned-subtree message marshals (no double free, 5x)"
else
bad "lang-41 cross-shard marshal" "$csm_bad"
fi
else
bad "lang-41 cross-shard marshal" "fixture did not build (need wovm-asan)"
fi
echo
echo "db-actor-accept: $((pass + fail)) checks, $fail failures"
[[ $fail -eq 0 ]] || exit 1

48
tests/regress/lang-41/cross-shard-marshal.wo Normal file
View file

@ -0,0 +1,48 @@
-- lang-41 phase C: a cross-shard message carrying an OWNED SUBTREE (a multi of
-- Text) sent AND called into an actor that lands on another shard. Pre-fix, the
-- payload's pointer was shared across arenas: the receiver and the sender's
-- graph both dropped the same multi/Text blocks -> a double free (ASan abort or
-- the settle livelock/hang). With the marshal fix each arena frees its own copy.
-- Needs WO_SHARDS>1 + the ASan build (scripts/db-actor-accept.sh drives it).
class Msg { tags: multi Text }
-- reads the owned subtree (forces the receiver to touch the crossed blocks),
-- then the message is dropped on the receiver's shard.
class Sink {
fn receive(msg: Msg) -> Int {
let n = len(msg.tags);
let i = 0;
let acc = 0;
while i < n { acc = acc + len(msg.tags[i]); i = i + 1; }
return acc;
}
}
-- churns cross-shard send + call, each carrying a fresh multi<Text> subtree.
class Client {
target: actor Msg
fn receive(msg: Msg) -> Int {
let j = 0;
while j < 8 {
let r = call(self.target, Msg { tags: ["a-${j}", "bb-${j}", "ccc-${j}"] });
send(self.target, Msg { tags: ["x-${j}", "yy-${j}"] });
j = j + 1;
}
return 0;
}
}
fn main() -> Int {
let sink: actor Msg = spawn Sink {};
let c1: actor Msg = spawn Client { target: sink };
let c2: actor Msg = spawn Client { target: sink };
let c3: actor Msg = spawn Client { target: sink };
let c4: actor Msg = spawn Client { target: sink };
send(c1, Msg { tags: ["go"] });
send(c2, Msg { tags: ["go"] });
send(c3, Msg { tags: ["go"] });
send(c4, Msg { tags: ["go"] });
print("dispatched");
return 0;
}