Shared UMEM and AF_XDP for Zero-Copy Extensible Data Planes using WebAssembly

2026-09-28

Why We Broke the WebAssembly Sandbox

We chose WebAssembly to enable safe, zero-downtime hot-reloading of dynamic network logic directly on the fast path, but doing this at scale requires zero-copy operations. Initially, we implemented WASM multi-memory to maintain strict sandbox isolation, but cross-memory access degraded performance by a factor of 2.

The solution was to break the rules: we dropped strict isolation, bypassed the Component Model, and mapped the shared AF_XDP UMEM straight into the upper region of the WASM linear memory alongside a private, per-instance metadata mailbox. This approach eliminates serialization penalties, enabling us to process 13.1 M packets per second—with payload inspection operating at just 99 cycles per packet—directly on the fast path. We are using an Intel E830-XXVDA2 NIC with 2x 25 Gbps ports, Intel Core™ i7-14700K, and Linux kernel 7.0.

The full code of the host-side platform and of some WASM plugin examples can be found at Wabrix.

Here is the exact memory architecture and address calculation that makes this zero-copy exchange possible.

The Wasm32 Linear Address Space

In a standard wasm32 environment, memory is a sandboxed, flat array of bytes managed by the host. Instead of keeping the packet buffer (UMEM) strictly on the host and copying packets into the Wasm sandbox one by one, we stitch a private memory region and a shared AF_XDP UMEM into a single contiguous 32-bit linear address space for each Wasm instance.

The Stitched Memory Map

Each Wasm instance receives a private linear-memory mapping backed by a memfd.

Private Lower Region (0 .. MAX_WASM_HEAP_BYTES): Contains the module's stack, heap, globals, and a private BatchMetadata mailbox at the top of the region.
Shared Upper Region (MAX_WASM_HEAP_BYTES .. 4 GiB): Directly maps the entire AF_XDP UMEM from the same pages used by xsk-core. All Wasm instances see the exact same physical packet frames through their respective UMEM alias.

 
 WASM Linear Memory Space -- Instance <n> (Up to 4 GiB)
 +----------------------------------------------------------------------+ 0x0000_0000
 |  Private Wasm Instance Region                                        |
 |  - Globals, Stack, and Heap (Rust allocator)                         |
 |                                                                      |
 +----------------------------------------------------------------------+ MAX_WASM_HEAP_BYTES - sizeof(BatchMetadata)
 |  Private BatchMetadata Mailbox                                       |
 +----------------------------------------------------------------------+ MAX_WASM_HEAP_BYTES (e.g. 2 GiB / 0x8000_0000)
 |  Shared AF_XDP UMEM Mapping                                          |
 |  - Same memfd-backed pages shared across all Wasm instances          |
 |  - 100% available for packet frames (no reserved control blocks)     |
 |                                                                      |
 +----------------------------------------------------------------------+ 4 GiB (32-bit max linear address space)

Absolute Address Translation

Because the shared UMEM is stitched above the private lower region (starting at MAX_WASM_HEAP_BYTES), calculating the guest pointer to a packet frame is a straightforward offset addition.

The host maintains packet descriptors relative to the native UMEM base (desc.addr). To convert a UMEM-relative descriptor to a guest linear address, the Wasm guest simply adds the base UMEM offset:


umem_offset = MAX_WASM_HEAP_BYTES
packet_wasm_addr = umem_offset + desc.addr

Zero-Copy Packet Payloads

Once packet_wasm_addr is calculated, the WASM instance uses standard pointer arithmetic (e.g., ptr::copy_nonoverlapping) to inspect or mutate headers and payloads directly within the shared UMEM. Writes go straight into the shared pages backed by the NIC's TX/RX ring buffers—zero serialization, zero heap allocations, and zero host-guest memory copying.

The Per-Instance Mailbox (Zero-Copy Metadata Exchange)

By default, the WebAssembly Component Model enforces a strict Canonical ABI. Returning structured metadata like (list<u32>, list<u32>) forces the guest to allocate heap memory and forces the host to perform boundary crossing copies.

To achieve 13M+ PPS throughput, we place a Per-Instance Metadata Mailbox (BatchMetadata) at the top of each instance's private lower memory region (MAX_WASM_HEAP_BYTES - sizeof(BatchMetadata)).

The Plugin Interface

The WIT interface between host and WASM instance is a lightweight signaling mechanism:


process-batch: func(batch-size: u32, umem-offset: u32, mailbox-offset: u32) -> u32;

batch-size: Number of valid PDU entries in the batch.
umem-offset: Base offset where UMEM is stitched (MAX_WASM_HEAP_BYTES).
mailbox-offset: Instance-local Wasm linear address of the BatchMetadata mailbox.

The `BatchMetadata` Layout

BatchMetadata is a repr(C) structure populated by the host and read/written by the plugin:


#[repr(C)]
pub struct BatchMetadata {
    pub pdus:    [Pdu0; MAX_BATCH_SIZE],  // Full packet descriptors (plugin view)
    pub actions: [u32;  MAX_BATCH_SIZE],  // Routing decisions written by plugin
}

Each Pdu0 slot is 64 bytes: a 56-byte WPdu core (header stack, descriptor address, length, options, metadata, timestamp) plus 8 tail bytes reserved for the host (*const SystemBus pointer):

 
 Byte layout of each PDU slot (64 bytes, repr(C, align(64))):
 +---------------------------------+-------------------+
 |  WPdu core  [0..56]             |  tail  [56..64]   |
 +---------------------------------+-------------------+
 |  header_stack (16)              |  host: SystemBus* |
 |  desc_addr (8)                  |  WASM: (unused)   |
 |  desc_len (4)                   |                   |
 |  desc_options (4)               |                   |
 |  metadata (16)                  |                   |
 |  timestamp (8)                  |                   |
 +---------------------------------+-------------------+

A compile-time assertion enforces binary layout alignment between host and guest:


const _: () = assert!(size_of::<HostBatch>() == size_of::<BatchMetadata>());

The Hot-Path Data Flow

Each batch execution follows four phases:

Populate (Host → Mailbox): A single SIMD-accelerated ptr::copy_nonoverlapping copies len PDU descriptors from host Batch memory into the guest's private mailbox.
WASM Execution: The host invokes process_batch(batch_size, umem_offset, mailbox_offset). The plugin reads descriptors, computes payload addresses as umem_offset + desc_addr, mutates frame headers in place, and writes routing decisions to actions[i].
Mutation (Mailbox → Host): A symmetric ptr::copy_nonoverlapping syncs the modified mailbox PDU descriptors back to the host Batch, leaving host tail pointers intact.
Routing: The host reads actions[i] and enqueues descriptors directly into the target AF_XDP TX rings or drop queues.

Raw Telemetry From an Example Run

Here is the raw telemetry log from a 500-million-packet pipeline run on an Intel 2x25G E830 NIC (Linux kernel 7.0) operating in NAPI busy poll mode. The two worker threads run on dedicated performance cores (3.417 GHz) on an Intel Core™ i7-14700K.

[2026-09-16T13:57:05Z INFO  wabrix_wasm::wasm_manager] WASM Node: generator_engine: generation run complete: packets=500,000,000, elapsed=38,095,137us, speed=13,125,034 packets/s
^C[2026-09-16T13:58:45Z INFO  wabrix] Stopping data plane workers...
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline] Worker 0: Shutdown complete
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline] Worker 0 pipeline 'generator_pipe' stopped:
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]   0 packets, 976,563 runs (595,562,273 idle, 976,563 timer-only)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]   38.065 CPU seconds load, 53.527 CPU seconds idle
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]     Node 0: generator_engine : 244 cycles/packet (35.766 total CPU seconds, 500,000,000 packets)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]     Node 1: tx_sink_0        : 36 cycles/packet (5.391 total CPU seconds, 500,000,000 packets)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]     Node 2: drop             : 0 packets processed
!!! KERNEL NOTIFY: Umem Group 0 is physically DEALLOCATING now !!!
!!! KERNEL NOTIFY: Umem Group 1 is physically DEALLOCATING now !!!
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline] Worker 1: Shutdown complete
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline] Worker 1 pipeline 'inspector_pipe' stopped:
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]   500,000,000 packets, 1,681,109 runs (26,661,295 idle, 0 timer-only)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]   38.043 CPU seconds load, 99.100 CPU seconds idle
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]     Node 0: rx_source        : 754 cycles/packet (110.413 total CPU seconds, 500,000,000 packets)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]       RX loaded runs: 98 cycles/packet (14.351 total CPU seconds)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]       RX empty runs: 96.062 total CPU seconds
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]       RX replenish_fill_queue: 4,938 avg cycles/call (0.578 total CPU seconds, 399,933 calls)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]     Node 1: rx_parser        : 42 cycles/packet (6.155 total CPU seconds, 500,000,000 packets)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]     Node 2: inspector        : 99 cycles/packet (14.544 total CPU seconds, 500,000,000 packets)
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]     Node 3: tx_sink_1        : 0 packets processed
[2026-09-16T13:58:45Z INFO  wabrix_lib::pipeline]     Node 4: drop             : 7 cycles/packet (1.110 total CPU seconds, 500,000,000 packets)