Hot-Reloading Stateful Network Logic at Line Rate Using WebAssembly

Updating Code While Packets Keep Arriving

WebAssembly plugins implement Wabrix's application-specific network behavior. They run as pipeline nodes that inspect and modify packet batches, choose routing actions, generate traffic, track protocol state, and respond to timers and control events. Updating a plugin changes the policy or protocol logic executed on the packet-processing path.

The Wabrix host provides AF_XDP packet I/O, shared packet memory, pipeline execution, and runtime lifecycle management. Plugins define the processing behavior and, when continuity is required, the persistent state layout. This makes the Wasm component the unit of deployment: new network logic can be installed while the host's workers, sockets, and frame ownership remain in service.

A network policy update has two continuity requirements: packets must keep moving, and the new code must pick up the state left by the old code. Losing a sequence counter is enough to corrupt loss measurements. Losing a connection table can break established sessions even when the packet queues stay intact.

During a reload, we replace the plugin's WebAssembly component while keeping its worker, pipeline, AF_XDP sockets, and packet memory running. Persistent plugin state lives in host-owned shared memory. The replacement attaches to those same pages, and the worker changes runtimes at a boundary between completed batches.

That design removes compilation, instance construction, state snapshotting, and runtime teardown from the handoff. It is how we make loss-free updates possible under load. Sustaining line rate still requires enough processing and queue headroom for the short handoff and any declared state upgrade.

A Small Pause Becomes a Large Packet Backlog

At 10 million packets per second, a two-millisecond interruption corresponds to 20,000 arriving packets. Rebuilding a runtime on a packet worker, waiting for a compiler, or destroying an old Wasmtime instance can turn an ordinary update into a burst of queue overflow.

There is a second problem with copying state. A snapshot taken while the old component continues processing is immediately stale. Pausing the old component to serialize its entire state introduces a handoff whose duration grows with that state.

Our reload path keeps the expensive preparation in the background and keeps the persistent bytes in place.

Keep the Packet-Processing Machinery Alive

The reload replaces the node's Wasmtime store and component bindings. The surrounding packet-processing machinery stays in service:

Packets already processed in an iteration finish with the old component. Work after activation uses the replacement. There is no need to close sockets, re-register UMEM, or move queued packets into another worker.

This is a component update within an existing graph. Changes that require a different worker layout, UMEM configuration, or rebuilt node graph belong to the full-restart path.

Prepare, Validate, Switch, Retire

We start one shared staging thread and one runtime reaper before packet workers enter their processing loops. Sending a reload request therefore does not create a new thread from a pinned worker.

Time --->

Packet worker   [old: process packets and timers................]
                                                          |
                                             runner boundary
                                             validate upgrade
                                             install candidate
                                                          |
                                                          [new: process...]

Staging thread     [load / compile / instantiate / init]--ready

Runtime reaper                                             [destroy old...]
 

The control plane compares the current and target configurations. A changed wasm_file_path becomes an UpdateConfiguration event addressed to the existing node. Its worker queues a staging job and continues processing with the active component.

The stager reads and compiles or loads the component, creates a fresh instance, maps its packet and state regions, and calls init-data. Compilation, instantiation, and initialization must succeed before that candidate can be activated. The component cache uses the artifact path and a SHA-256 content hash.

The background threads use the controller's allowed CPU affinity and normal SCHED_OTHER scheduling. They do not inherit a packet worker's restricted single-core mask or real-time policy. That gives the scheduler more freedom; deployments still need CPU and memory headroom for staging.

Each Wasm node has a one-job staging mailbox. A newer queued request replaces its stale predecessor, and generation counters prevent a superseded result from being activated. An already-running build may finish, but its obsolete candidate is discarded off the packet-processing thread.

The runner checks completed candidates during its existing timer and control polling. It does not add a reload check to every packet's processing path. While a candidate is being prepared, the old runtime continues serving both packets and timers.

State Continuity Comes From Shared Pages

One logical plugin node on one worker owns one dedicated state region. Both its active instance and its replacement map the same memfd-backed pages:

  +---------------------------+     +---------------------------+
  | Active Wasm instance      |     | Staged Wasm instance      |
  | Private heap and mailbox  |     | Private heap and mailbox  |
  +-------------+-------------+     +-------------+-------------+
                |                                 |
                +----------------+----------------+
                                 |
                   +-------------v-------------+
                   | Same per-node state pages |
                   | Header + fixed payload    |
                   +---------------------------+

  During staging: only the active instance accesses the live payload.
  After handoff:  only the replacement accesses the live payload.
 

The shared region is separate from packet UMEM and consumes no packet frames. Other nodes and workers receive their own state regions. In the guest's stitched linear memory, the region follows the page-aligned UMEM alias:

state_offset   = 2 GiB + align_up(umem_size, 64 KiB)
payload_offset = state_offset + 64
 

The 64-byte header records the state type ID, schema version, payload size, reserved capacity, and alignment. The current host provides a maximum payload capacity of 127 KiB per reloadable node, within a 128 KiB mapping including header and padding.

The plugin declares its layout through shared-state.attach; the host checks it before returning a payload offset. The SDK's SharedState<T> wrapper checks the typed view's size and alignment. A fresh instance initializes its payload once. A replacement attaches to the existing payload without clearing it.

The important consequence is that state cannot become stale while the replacement is compiling. The old instance keeps updating the actual pages that the new instance will use. There is no snapshot to catch up or restore.

The single-writer rule is a plugin contract. Staged initialization must leave the active payload alone, and a candidate requiring an upgrade must not access its target layout before the upgrade callback runs.

What the Inspector Preserves

The inspector example puts its bypass setting, packet and byte counters, latency histogram, expected UDP sequence number, and sequence anomaly counters in the shared payload. Reloading its code therefore keeps both the counters and the history needed to interpret the next packet.

Its new private runtime still initializes configuration such as MAC addresses and timing calibration. Persistent operational state and runtime-local configuration have different lifetimes.

Other stateful plugins must explicitly adopt this contract. State left only in a guest's private heap is lost when that runtime is retired. Connection tables and timer state need a plugin-defined persistent representation and must fit within the reserved capacity; existing plugin code does not acquire state continuity merely by being reloadable.

A Short Handoff With One Active Writer

When the runner observes a ready candidate, the current batch has finished and the old instance is no longer executing plugin code. For a matching schema, the persistent payload is already usable. The host swaps the store and bindings into the existing WasmEngine node.

The old runtime is then submitted to the reaper with a nonblocking send. If the reaper queue is full, the runner retains it for a later attempt. Wasmtime's destruction work stays off the packet-processing thread.

This keeps the handoff independent of copying the persistent payload. There is still control-path work at the boundary, and an upgrade callback adds to it. Keeping that work short is part of achieving loss-free operation at high load.

Evolving State Without Losing Its Existing Meaning

Code can change while its state layout stays the same. When the layout must grow, we support declared append-only upgrades. For example:

#[repr(C)]
#[derive(Clone, Copy)]
struct StateV1 {
    packets_inspected: u64,
    next_sequence: u64,
}

#[repr(C)]
#[derive(Clone, Copy)]
struct StateV2 {
    previous: StateV1,
    policy_matches: u64,
}
 

Here the original 16 bytes retain their offsets, types, and meaning. The replacement adds one eight-byte field and declares which earlier schema it can extend. Its requested layout must fit the region's existing declared capacity; page padding cannot be claimed as extra payload space.

At the runner boundary, the host calls:

upgrade-shared-state: func(
    source-schema-version: u32,
    source-payload-size: u32
) -> result<_, string>;
 

The callback initializes only the appended fields. After it succeeds, the host commits the new schema version and payload size, then activates the candidate. If it fails, the header stays unchanged and the old runtime remains active. That failure behavior depends on the callback preserving the old payload prefix; the host cannot infer field meaning from byte sizes.

Reordering fields, changing their meaning, or rebuilding a large table is outside this append-only contract. Upgrade work also runs on the worker at handoff, so its duration must fit the deployment's queue and latency budget.

Preserving the Pipeline's Runtime State Matters Too

State continuity extends beyond the plugin payload. A configuration update must also preserve the current lifecycle of unrelated pipelines.

For example, Running delay=2 expresses a two-millisecond startup delay. Once the pipeline is running, applying a new inspector artifact must not translate that unchanged setting back into Paused. Doing so would stop the generator's timers, accumulate generation credit, and produce catch-up bursts after the delay.

Reconciliation therefore publishes RCU changes only for affected pipelines and nodes. It updates lifecycle fields only when the configured lifecycle actually changes. Regression tests protect both an already-completed startup delay and a pending startup deadline.

Applying an Update

Deploy the candidate at a new, immutable artifact path. In the existing node's target configuration, change its effective Wasm path, for example:

hot_args = { wasm_file_path = "/opt/wabrix/inspector-v2.wasm" }
 

Then use the running CLI:

apply /etc/wabrix/inspector-v2.toml
 

Keep the node graph and infrastructure unchanged for a component-only update. Replacing file contents at the existing path does not itself create a configuration difference. Watch the staging and activation messages for each worker: accepting a configuration request is earlier than completing its asynchronous activation.

Each worker switches independently. A rollout can briefly contain both code versions, so cross-worker behavior must tolerate that overlap. This mechanism does not provide a simultaneous, globally atomic policy switch.

What Happens When Preparation Fails?

These are preparation and handoff guarantees. A successfully activated version can still contain a packet-processing bug. The reload mechanism does not automatically restore a retired runtime after a later trap, and shared memory is not a durable checkpoint across a process crash or full data-plane restart.

Demonstrating Zero Loss Under Load

Keeping the worker alive removes a major source of interruption, but finite queues still set a budget. A useful first estimate is:

buffering_time = available_packet_slots / incoming_packets_per_second
 

The handoff, any upgrade callback, and scheduling delays must fit comfortably within the actual available buffering and service-rate headroom. The budget depends on the current RX backlog, fill-ring supply, UMEM ownership, downstream TX pressure, and workload. Configured ring capacity alone is not free space.

Line rate also depends on packet size and link speed. A packets-per-second measurement must be tied to the tested traffic and wire rate before it can be called a line-rate result.

Our memory architecture article includes the recorded 500-million-packet, approximately 13.1 Mpps run. That is a throughput baseline. Demonstrating a loss-free reload requires a measurement that spans the update itself:

1. Establish continuous traffic with identifiable streams and monotonic sequence numbers. Keep stream-to-worker assignment stable.
2. Record NIC, socket, TX, and inspector counters before applying the update.
3. Change the artifact while traffic continues, and capture activation times for every worker.
4. Confirm packet accounting, inspect sequence gaps and reordering separately, and verify that persistent counters and expected sequences did not reset.
5. Compare latency distributions around the handoff, then repeat with a rejected candidate and a declared append-only upgrade.

An activation log establishes that code changed. Continuous sequence tracking, frame accounting, preserved application state, and latency measurements together establish what happened to the traffic during that change.

Implementation and Contracts

The implementation is available in Wabrix, with plugin bindings and examples in wabrix_api. The relevant pieces are the reload controller and background service, the engine handoff, and the shared-state contract.