Shared UMEM and AF_XDP for Zero-Copy Extensible Data Planes using WebAssembly
Why We Broke the WebAssembly Sandbox
We chose WebAssembly to enable safe, zero-downtime hot-reloading of dynamic network logic directly on the fast path, but doing this at scale requires zero-copy operations. Initially, we implemented WASM multi-memory to maintain strict sandbox isolation, but the cross-memory access degraded performance by a factor of 2.
The solution was to break the rules: we dropped strict isolation, bypassed the Component Model, and mapped a shared AF_XDP UMEM straight into the WASM linear memory and used a custom per-instance metadata mailbox. This approach eliminates serialization penalties, enabling us to process 12.9M packets per second—with payload inspection operating at just 101 cycles per packet—directly on the fast path. We are using an Intel E830-XXVDA2 NIC with 2x 25 Gbps ports, Intel Core™ i7-14700K, and Linux kernel 7.0.
The full code of the host-side platform and of some WASM plugin examples can be found at Wabrix.
Here is the exact memory architecture and address calculation that makes this zero-copy exchange possible.
The Wasm32 Linear Address Space
In a standard wasm32 environment, memory is not physical RAM; it is a sandboxed, flat array of bytes managed by the host.
Instead of keeping the packet buffer (UMEM) on the host and copying packets into the Wasm sandbox one by one, we let the host map an entire physical UMEM directly into the Wasm linear memory.
The Unified Memory Map
Host-Side Physical UMEM Layout
The host maps the AF_XDP UMEM as a contiguous physical region divided into three zones into the WebAssembly address room. The three zones are:
- partitions for each WebAssembly instance, used by the instance as main memory for globals, stack, and the heap
- a control block with a "mailbox" for each WebAssembly instance
- the UMEM payload frames used by the AF_XDP Rx, Tx, Fq and Cq rings.
Host Physical UMEM (byte offsets from UMEM base ptr)
+----------------------------------------------------------------------+
| Main memory partitions [0 .. reserved_bytes - control_block_bytes) |
| |
| partition_start_bytes_0 = 0 |
| +--------------------------------------+ |
| | Instance 0: Stack, Heap, Globals | (partition_size bytes) |
| +--------------------------------------+ |
| partition_start_bytes_1 = 1 * partition_size |
| +--------------------------------------+ |
| | Instance 1: Stack, Heap, Globals | (partition_size bytes) |
| +--------------------------------------+ |
| partition_start_bytes_N = N * partition_size |
| +--------------------------------------+ |
| | Instance N: Stack, Heap, Globals | (partition_size bytes) |
| +--------------------------------------+ |
| |
+----------------------------------------------------------------------+
| Control Block [reserved_bytes - ctrl_block_bytes .. reserved_bytes) |
| |
| mailbox_offset_0 = reserved_bytes - 1 * sizeof(BatchMetadata) |
| mailbox_offset_1 = reserved_bytes - 2 * sizeof(BatchMetadata) |
| mailbox_offset_N = reserved_bytes - (N+1) * sizeof(BatchMetadata) |
| |
+----------------------------------------------------------------------+ <-- reserved_bytes
| AF_XDP Payload Frames [reserved_bytes .. total_size) |
| |
| Raw Ethernet frames from / to the NIC (shared by all instances) |
| |
+----------------------------------------------------------------------+
Plugin-Side WASM Linear Memory View (Instance )
For this mapping the host uses a WebAssembly runtime like Wasmtime. The partition_start_bytes is the byte offset from the UMEM base where this
WebAssembly instance's WASM address 0 begins. The window extends from that offset to the end of the full UMEM, so the plugin can access both its own scratchpad
and all packet frames above the reserved region. Wasmtime maps address 0 of the WASM linear memory for instance partition_start_bytes_<n> on the host. The window extends to the end of the full UMEM. From the plugin's perspective:
WASM Linear Memory -- Instance <n> (WASM address 0x0000_0000 onward)
+----------------------------------------------------------------------+
| WASM address 0x0000_0000 |
| |
| Static Data / Stack / Heap |
| (Rust allocator stays within this partition -- software convention) |
| |
+----------------------------------------------------------------------+
| Instance <n>'s BatchMetadata Mailbox |
| WASM address = mailbox_offset_<n> - partition_start_bytes_<n> (> 0) |
| |
+----------------------------------------------------------------------+
| AF_XDP Payload Frames |
| WASM address = desc_addr - partition_start_bytes_<n> (> 0) |
| |
+----------------------------------------------------------------------+
Because the Control Block sits at the end of the reserved region (above every instance's partition_start_bytes), and AF_XDP frames start beyond that, both the mailbox and every packet frame are at positive WASM addresses within the linear memory window.
Absolute Address Translation
Because the UMEM partition containing the packets is mapped with a specific offset inside the Wasm memory, the Wasm plugin cannot assume the UMEM starts at address 0. It must translate the relative UMEM byte offsets provided by AF_XDP into absolute Wasm memory pointers.
partition_offset: The host tells the Wasm plugin the partition-offset of its WASM addr 0 with respect to the UMEM start.wrapping_sub: Subtracting thepartition_offsetfrom0creates a negative UMEM base address (in 32-bit unsigned math)umem_addr. This is added by the WASM code to an UMEM frame address to get the corresponding absolute WASM address. This allows the WASM instance to use the exact same code the host uses to transform a UMEM-relative byte offset into an absolute host address.
The following diagram shows the relationship for one UMEM frame. desc.addr is a UMEM-relative byte offset to the packet data (and may include headroom);
the frame's aligned start is obtained by clearing the low log2(frame_size) bits.
In the guest, adding desc.addr to umem_addr is equivalent to subtracting partition_offset from the UMEM-relative byte offset (desc.addr):
Host UMEM byte offsets (relative to the native UMEM base)
0 ------------------------------------------------------------> total_umem_size
| ^ |
| | |
| partition_start_bytes_<n> == partition_offset |
| (WASM address 0 for instance <n>) |
| | |
| | |
| | frame-aligned start |
| | = desc.addr & ~(frame_size - 1) |
| | | |
| | v |
| | +------------------------------+ |
| | | UMEM frame | |
| | | headroom | packet data | |
| | | ^ | |
| | | desc.addr | |
| | +------------------------------+ |
| | | | |
+--------------------------------------------------------------------+
| |
| | WASM linear memory for instance <n> (32-bit addresses)
| 0 -------------------------------------------------------->
^
| umem_addr = 0u32.wrapping_sub(partition_offset)
^
| frame base = umem_addr + frame-aligned start
= frame-aligned start - partition_offset
^
| packet data = umem_addr + desc.addr
= desc.addr - partition_offset
For example, with partition_offset = partition_start_bytes_<n> = 0x2000 and desc.addr = 0x5000, umem_addr = 0xffffe000 and the packet data is at WASM address 0x3000 (0xffffe000 + 0x5000, modulo 32 bits).
Zero-Copy Packet Payloads
Once the absolute pointer to a packet is calculated, the WASM instance uses standard pointer arithmetic (e.g., ptr::copy_nonoverlapping) to write headers and payloads directly into the shared UMEM. This way the Wasm engine is writing the bytes directly into the NIC's TX ring buffer. No serialization, no Vec allocation, and no boundary crossing.
The Per-Instance Mailbox (Zero-Copy Metadata Exchange)
By default, the WebAssembly Component Model enforces a strict Canonical ABI. Returning data like (list<u32>, list<u32>) from the guest forces the guest to allocate heap memory, and forces the host to allocate native vectors and physically copy the data across the sandbox boundary.
To achieve 10M+ PPS throughput, we bypass the Component Model entirely using a Per-Instance Metadata Mailbox located within the shared Control Block at the end of the reserved UMEM range.
To prevent the physical NIC hardware from overwriting the Control Block with incoming Ethernet frames, the Host Runner utilizes the AF_XDP Fill Queue (FQ). The host withholds all frames belonging to the reserved UMEM range from the Fill Queue during initialization. Because the NIC never receives these addresses, it will never DMA packets into them.
The Plugin Interface
The WIT interface between host and WASM instance reduces to a minimal signaling mechanism — no data crossing the boundary:
process-batch: func(batch-size: u32, partition-offset: u32, mailbox-offset: u32) -> u32;
batch-size: Number of valid PDU entries written into the mailbox.partition-offset: Equalspartition_start_bytes_<n>used to computeumem_addr.mailbox-offset: UMEM-relative byte offset of this instance'sBatchMetadata. Equalsn × sizeof(BatchMetadata).
The BatchMetadata Layout
BatchMetadata is the shared control structure written by the host and read/written by the plugin:
#[repr(C)]
pub struct BatchMetadata {
pub pdus: [Pdu0; MAX_BATCH_SIZE], // Full packet descriptors (plugin view)
pub actions: [u32; MAX_BATCH_SIZE], // Routing decisions written by plugin
}
Each Pdu0 slot is 64 bytes: a 56-byte WPdu core (header stack, descriptor address, length, options, metadata, timestamp) plus 8 bytes the plugin must never read or write. The host uses an identical-layout HostBatch type that places a *const SystemBus pointer in those last 8 bytes:
Byte layout of each PDU slot (64 bytes, repr(C, align(64))):
+---------------------------------+-------------------+
| WPdu core [0..56] | tail [56..64] |
+---------------------------------+-------------------+
| header_stack (16) | host: SystemBus* |
| desc_addr (8) | plugin: (unused) |
| desc_len (4) | |
| desc_options (4) | |
| metadata (16) | |
| timestamp (8) | |
+---------------------------------+-------------------+
A compile-time assertion enforces layout compatibility:
const _: () = assert!(size_of::<HostBatch>() == size_of::<BatchMetadata>());
The Hot-Path Data Flow
Each call to execute_wasm_and_route follows four phases:
Phase 1 — Populate (Host → Mailbox):
A single ptr::copy_nonoverlapping copies all len Pdu structs from the host Batch into the UMEM mailbox. LLVM lowers this to SIMD instructions, moving ~16 KB in a handful of CPU cycles. The plugin sees the full packet descriptor (address, length, header metadata) for each frame.
ptr::copy_nonoverlapping(
batch.as_ptr() as *const Pdu,
(*mailbox_ptr).pdus.as_mut_ptr(),
len,
);
Phase 2 — WASM Execution:
The host calls process_batch(batch_size, partition_offset, mailbox_offset). The plugin reads pdus[i] for packet metadata, processes frames zero-copy, and writes its routing decision to actions[i]. It may also mutate WPdu fields such as desc_len or desc_addr to reflect packet modifications.
Phase 3 — Mutation (Mailbox → Host):
A symmetric ptr::copy_nonoverlapping copies the plugin-modified PDU slots back into the host Batch. The system_bus tail bytes are preserved unchanged (plugins never touch them), so the host's Pdu state remains consistent.
ptr::copy_nonoverlapping(
(*mailbox_ptr).pdus.as_ptr(),
batch.as_mut_ptr() as *mut Pdu,
len,
);
Phase 4 — Routing:
The host reads actions[i] from the mailbox and distributes PDUs across output Batch queues according to each plugin's routing decision.
This architecture guarantees that:
- Both packet payloads and control metadata are 100% zero-copy and zero-allocation on the hot path.
- Plugin instances execute concurrently without contention: each instance owns an exclusive, non-overlapping
BatchMetadataslice within the Control Block. - The control channel is hardware-protected: withheld FQ frames ensure the NIC never overwrites the reserved UMEM range.
Raw Telemetry From an Example Run
To demonstrate the real-world overhead, here is the raw telemetry log from a 500-million-packet pipeline using an Intel 2x25G E830 NIC (Linux kernel 7.0) in NAPI busy poll mode. The two workers driving the UDP generator-pipe and the inspector-pipe run each on a separate performance core at 3.417 GHz on an Intel i7-14700K.
[2026-09-12T20:44:46Z INFO wabrix_wasm::wasm_manager] WASM Node: generator_engine: generation run complete: packets=500,000,000, elapsed=38,487,359us, speed=12,991,278 packets/s
^C[2026-09-12T20:44:56Z INFO wabrix] Stopping data plane workers...
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Worker 0: Shutdown complete
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Worker 0 pipeline 'generator_pipe' stopped:
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] 0 packets, 976,565 runs (59,599,611 idle, 976,565 timer-only)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] 38.435 CPU seconds load, 5.894 CPU seconds idle
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Node 0: generator_engine : 221 cycles/packet (32.466 total CPU seconds, 500,000,000 packets)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Node 1: tx_sink_0 : 36 cycles/packet (5.400 total CPU seconds, 500,000,000 packets)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Node 2: drop : 0 packets processed
!!! KERNEL NOTIFY: Umem Group 0 is physically DEALLOCATING now !!!
!!! KERNEL NOTIFY: Umem Group 1 is physically DEALLOCATING now !!!
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Worker 1: Shutdown complete
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Worker 1 pipeline 'inspector_pipe' stopped:
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] 500,000,022 packets, 1,755,276 runs (2,826,911 idle, 0 timer-only)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] 38.423 CPU seconds load, 10.795 CPU seconds idle
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Node 0: rx_source : 169 cycles/packet (24.819 total CPU seconds, 500,000,022 packets)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] RX loaded runs: 98 cycles/packet (14.390 total CPU seconds)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] RX empty runs: 10.429 total CPU seconds
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] RX replenish_fill_queue: 4,165 avg cycles/call (0.489 total CPU seconds, 400,762 calls)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Node 1: rx_parser : 41 cycles/packet (6.082 total CPU seconds, 500,000,022 packets)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Node 2: inspector : 101 cycles/packet (14.805 total CPU seconds, 500,000,022 packets)
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Node 3: tx_sink_1 : 0 packets processed
[2026-09-12T20:44:56Z INFO wabrix_lib::pipeline] Node 4: drop : 7 cycles/packet (1.157 total CPU seconds, 500,000,022 packets)