fix(ffi): a refused allocation answers null instead of ending the process

dev_alloc built its buffer with vec!, which allocates on the path that cannot
fail: a valid layout the allocator refuses goes to handle_alloc_error, and that
aborts. An abort is not a panic, so the catch_unwind around it never ran.
dev_alloc(isize::MAX) died on SIGABRT, taking every other session in the
process with it, and the same call into dev.wasm trapped on unreachable — under
a doc and a header that both promised null.

Ask the allocator directly. Layout::from_size_align refuses a length with no
layout, alloc_zeroed answers null when there is no memory, and nothing left in
there can panic, so the catch_unwind goes. dev_release frees the buffer as the
boxed [u8] whose layout this is.

The module doc and dev.h now say how far "the process stays up" reaches: a
panic, and no further. Memory the library needs for itself still aborts when
refused — with the module's memory lent out, a dev_step on a 512 MiB input
traps — and dev_alloc is the one place a refusal comes back.

The test asks for isize::MAX bytes and the two lengths past it; against the old
body it kills the test binary with SIGABRT, and so does the C fixture asking
for PTRDIFF_MAX through dev.h. The wasm import list is unchanged.

Co-authored-by: Hanzo Dev <[email protected]>

crates/ffi/include/dev.h

@@ -15,6 +15,9 @@
  *     left for its answer to belong to.
  *   - A call that panics answers DEV_PANIC and poisons that session; its later
  *     calls answer DEV_POISON until dev_drop. Other sessions are untouched.
+ *   - A refused allocation is not a panic. Memory the library needs for itself
+ *     and is refused ends the process, or in wasm traps the instance; dev_alloc
+ *     is the one place a refusal comes back, as a null.
  *   - A dev_buf is written only on DEV_OK. On any other status the out
  *     parameter is left as the caller passed it.
  *

crates/ffi/src/lib.rs

@@ -21,9 +21,15 @@
 //!   status to answer with runs inside `catch_unwind`; an escaped panic answers
 //!   [`DEV_PANIC`] and poisons that session, whose every later call answers
 //!   [`DEV_POISON`] until it is dropped. The process stays up and the other
-//!   sessions are untouched. [`dev_abi`] is the exception and has no status
-//!   channel to answer on: it reads a constant, so there is nothing in it to
-//!   panic.
+//!   sessions are untouched. [`dev_abi`] and [`dev_alloc`] are the exceptions
+//!   and have no status channel to answer on: one reads a constant and the
+//!   other asks the allocator, whose refusal is a null, so there is nothing in
+//!   either to panic.
+//! - **A refused allocation is not a panic.** Rust aborts when memory it needs
+//!   for itself is refused — the copy of an input, a transcript that grew — and
+//!   `catch_unwind` never sees an abort: natively it ends the process, in wasm
+//!   it traps the instance. [`dev_alloc`] is the one place a refusal comes
+//!   back, as a null, because there the host asked and can check.
 
 mod slab;
 
@@ -248,10 +254,15 @@ pub extern "C" fn dev_alloc(len: usize) -> *mut u8 {
     if len == 0 {
         return std::ptr::null_mut();
     }
-    match catch_unwind(|| Box::into_raw(vec![0_u8; len].into_boxed_slice()).cast::<u8>()) {
-        Ok(ptr) => ptr,
-        Err(_) => std::ptr::null_mut(),
-    }
+    // The allocator is asked directly. A `vec!` answers a refusal by aborting
+    // the process, which is not a panic, so no `catch_unwind` ever sees it; a
+    // length with no layout at all is refused here instead.
+    let Ok(layout) = std::alloc::Layout::from_size_align(len, 1) else {
+        return std::ptr::null_mut();
+    };
+    // SAFETY: `len` is not zero, so neither is the layout. [`dev_release`]
+    // frees this as a boxed `[u8]` of the same length, whose layout this is.
+    unsafe { std::alloc::alloc_zeroed(layout) }
 }
 
 /// Take back a buffer [`dev_alloc`] lent.
@@ -676,6 +687,18 @@ mod tests {
         unsafe { dev_release(ptr, 64) };
     }
 
+    /// No allocator has `isize::MAX` bytes, and the layout for them is still a
+    /// valid one: the refusal comes from the allocator, which is the refusal
+    /// that aborts a process rather than unwinding. The two lengths past it
+    /// have no layout at all. Every one of them is a null the host can check.
+    #[test]
+    fn a_refused_allocation_is_null_rather_than_an_abort() {
+        let most = isize::MAX.unsigned_abs();
+        for len in [most, most + 1, usize::MAX] {
+            assert!(dev_alloc(len).is_null(), "{len} bytes were lent");
+        }
+    }
+
     /// Asking for nothing lends nothing, and giving nothing back is not a fault.
     #[test]
     fn nothing_is_lent_for_nothing() {

crates/ffi/tests/abi.c

@@ -117,6 +117,13 @@ int main(void) {
   CHECK(dev_step(0, (const uint8_t *)TURN, strlen(TURN), &refused) == DEV_HANDLE);
   CHECK(dev_step(session, (const uint8_t *)TURN, strlen(TURN), NULL) == DEV_NULL);
 
+  /* What dev_alloc lends comes back through dev_release, and an allocation no
+   * allocator can make is a null rather than the end of the process. */
+  uint8_t *lent = dev_alloc(16);
+  CHECK(lent != NULL);
+  dev_release(lent, 16);
+  CHECK(dev_alloc(PTRDIFF_MAX) == NULL);
+
   if (failures != 0) {
     fprintf(stderr, "%d check(s) failed\n", failures);
     return 1;