Handling Tails and Alignment in SIMD Loops

This page answers one task: your SIMD loop processes 16 bytes (or 4 floats) per iteration, but real inputs come in arbitrary lengths and at arbitrary addresses. You want the leftover elements — the tail — handled correctly and cheaply, and you want to know whether alignment matters for WebAssembly SIMD.

Prerequisites

  • [ ] A SIMD kernel in Rust (core::arch::wasm32) or C (wasm_simd128.h) with a scalar reference.
  • [ ] A test harness that can run many input lengths quickly.
  • [ ] A SIMD-enabled build target.

Why tails exist

A vector loop handles W elements per iteration (16 bytes, 8 i16, 4 f32). For a length n, it covers n − n % W elements; the last n % W — between 0 and W − 1 — are left over. Every SIMD loop needs a plan for them, and most SIMD bugs live in that plan: off-by-one bounds that skip the last element, process one twice in a way that matters, or read beyond the end of the buffer.

There are four standard strategies, and the right one depends on whether the operation is idempotent, whether the buffer can be padded, and how short typical inputs are.

Four ways to handle a SIMD tail A scalar epilogue processes leftovers one at a time and works everywhere. An overlapping final vector reprocesses some elements and works when the operation is idempotent per element. Padding the buffer lets the vector loop run past the end safely. Blending with bitselect writes only the valid lanes of a final vector when padding is available for loads. strategy works when cost typical use scalar epilogue always up to W−1 scalar steps default overlapping last vector n ≥ W, idempotent per element one extra vector maps, filters padded buffer you own the allocation padding bytes image rows, owned arrays blend with bitselect loads may overrun safely mask setup short inputs

Step 1 — the scalar epilogue

use core::arch::wasm32::*;

pub fn add_one(data: &mut [u8]) {
    let n = data.len();
    let mut i = 0;
    unsafe {
        while i + 16 <= n {
            let p = data.as_mut_ptr().add(i) as *mut v128;
            v128_store(p, u8x16_add(v128_load(p), u8x16_splat(1)));
            i += 16;
        }
    }
    for x in &mut data[i..] { *x = x.wrapping_add(1); }     // 0–15 leftover bytes
}

The condition i + 16 <= n (not i < n) is the essential detail: it guarantees each vector lies entirely inside the slice. The epilogue reuses the scalar reference logic, so it is correct by construction. For short inputs (under 16 bytes) the vector loop does nothing and the scalar loop does everything — fine, as long as short inputs are not the common case.

Step 2 — overlap the last vector

When processing an element twice gives the same result — writing f(x) to a separate output, or searching for a byte — handle the tail with one more vector that ends exactly at the end of the data:

if n >= 16 && i < n {
    let j = n - 16;                                  // overlaps the previous vector
    let src = v128_load(input.as_ptr().add(j) as *const v128);
    v128_store(output.as_mut_ptr().add(j) as *mut v128, transform(src));
}

This replaces up to 15 scalar steps with one vector operation. It is wrong for in-place updates that are not idempotent: add_one applied in place would add 2 to the overlapping bytes. Use it for out-of-place transforms, comparisons and searches (taking care not to report a match twice).

Scalar epilogue versus overlapping final vector A scalar epilogue processes leftover elements one at a time and is always correct. An overlapping final vector processes the last 16 elements in one step, recomputing some, which is faster but only correct when recomputation produces the same result. scalar epilogue correct for any operation up to W−1 scalar steps handles n < W safe default overlapping last vector one extra vector step needs n ≥ W idempotent ops only out-of-place transforms

Step 3 — pad buffers you own

If you allocate the buffer, round its capacity up to a multiple of 16 (plus 16 for loops that read ahead) and let the vector loop run past the logical end. The extra lanes compute garbage that nobody reads. This removes tail code entirely and is standard for image rows, audio frames and internal scratch arrays. Never apply it to slices you do not own — the bytes after them belong to other data.

Step 4 — understand alignment in Wasm

v128.load and v128.store work at any address in WebAssembly; the alignment immediate is only a hint, and a misaligned access is never a trap. On current x64 and Arm hardware, unaligned vector loads that do not cross a cache line run at full speed, and those that do cost a little extra. In practice, aligning buffers to 16 bytes gives a small, measurable gain for streaming loops, and nothing more. Two ways to get it: allocate with 16-byte alignment (Rust’s #[repr(align(16))], or a Layout with align 16), or process a short scalar prologue until the pointer is aligned and then run aligned vectors — rarely worth the extra code in Wasm.

Step 5 — test every length

Tail bugs hide at specific lengths. Test every length from 0 to at least 3W + 1 (0–49 for byte loops), at several starting offsets (0–15) inside a larger buffer, and compare with the scalar reference. Fill the bytes around the slice with a sentinel and assert they are unchanged after the call, which catches writes past the end that a value comparison would miss.

for off in 0..16 { for len in 0..50 {
    let mut buf = vec![0xAAu8; 80];
    let mut expect = buf.clone();
    add_one(&mut buf[off..off + len]);
    for x in &mut expect[off..off + len] { *x = x.wrapping_add(1); }
    assert_eq!(buf, expect, "off={off} len={len}");      // sentinels must be untouched
}}

Reading past the end safely

Some fast kernels load a whole vector even when fewer bytes remain and ignore the extra lanes. In native code this can fault at a page boundary; in Wasm, any in-bounds address of linear memory is readable, so the load only traps at the very end of memory. That makes over-reading usually safe in Wasm — but it reads other data, which must not influence the result, and it can trap if the buffer sits at the end of memory. Prefer padded buffers or an explicit tail over relying on this.

Blending the final vector with bitselect

WebAssembly SIMD has no masked store, but a masked result is easy: compute a mask whose lanes are all-ones for valid elements and zero for the rest, and use v128.bitselect to keep the new value only in valid lanes, leaving the original bytes elsewhere. This requires that loading and storing a full vector at the tail is in bounds — true for padded buffers, or for a slice that is followed by other bytes you control.

let rem = n - i;                                         // 1..=15 valid bytes
let idx = u8x16(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15);
let mask = u8x16_lt(idx, u8x16_splat(rem as u8));        // all-ones for lanes < rem
let p = data.as_mut_ptr().add(i) as *mut v128;
let old = v128_load(p);
v128_store(p, v128_bitselect(u8x16_add(old, u8x16_splat(1)), old, mask));

Lanes beyond rem are written back with their own original values, so the bytes after the logical end are unchanged. Unlike an overlapping vector, this works for in-place, non-idempotent updates, and unlike a scalar epilogue it costs a fixed handful of instructions regardless of the remainder. Its limitation is concurrency: if another thread writes the bytes past the end between the load and the store, the store overwrites that write — so do not use it on shared memory regions that others modify.

Choosing a strategy in practice

For most code, start with the scalar epilogue: it is obviously correct and its cost disappears for inputs longer than a few hundred elements. Move to an overlapping vector when profiles show many short inputs and the operation is out of place. Pad buffers when you own the allocation anyway — image rows, audio frames, decoder scratch space — because it removes tail logic from every kernel at once. Reserve bitselect blending for in-place kernels on padded memory where short inputs dominate. Whatever you choose, write the reason in a comment next to the loop, because the next person to change the bounds will need it.

Tails in multi-accumulator loops

Unrolled loops that process two or four vectors per iteration for latency hiding have two tails: the leftover whole vectors (fewer than the unroll factor) and the leftover elements within the last vector. Handle the first with a single-vector loop and the second with one of the strategies above, and remember to combine all accumulators before the final reduction.

Expected output

The SIMD loop uses i + W <= n bounds; leftovers go through a scalar epilogue or an overlapping final vector where safe; owned buffers are padded; tests over lengths 0–49 and offsets 0–15 pass with sentinels intact; and alignment is treated as a minor optimisation rather than a requirement.

Gotchas

  • Bounds written as i < n. The last vector runs past the end. Use i + W <= n.
  • Overlapping tails on in-place updates. Elements change twice. Use a scalar tail.
  • Padding slices you do not own. Overwrites neighbouring data.
  • Testing only round lengths. Tail bugs hide. Test 0 to 3W + 1.
  • Assuming misalignment traps. It never does in Wasm; it is only slightly slower.
  • Bitselect tails on shared memory. The full-vector store can overwrite another thread’s writes. Use a scalar tail there.

Performance note

For 37-byte inputs processed millions of times, replacing a 5-step scalar epilogue with an overlapping final vector cut time per call by about 30%; for 4 KB inputs, the tail strategy made no measurable difference.

Time per call for short inputs by tail strategy Nanoseconds per call for 37-byte inputs with a scalar epilogue, an overlapping final vector, and a padded buffer that needs no tail handling. ns per call (37-byte input) scalar epilogue 23 ns overlapping last vector 16 ns padded buffer 15 ns

Frequently Asked Questions

Is there a masked load in Wasm SIMD? No — emulate it with padding, overlapping loads, or v128.bitselect on the result.

Does the alignment immediate change behaviour? No — it is a hint; behaviour is identical for any address.

Should short inputs skip SIMD entirely? Often yes — below about 2W elements, a scalar loop can be as fast and simpler.

Do compilers generate tails automatically? Yes — autovectorized loops include scalar epilogues; hand-written intrinsics need your own.

Can bitselect replace a masked store? Yes, when a full-vector load and store at the tail is in bounds and no other thread writes those bytes.

How many tails does an unrolled loop have? Two — leftover whole vectors and leftover elements; handle each separately.

← Back to Wasm SIMD & Vectorized Computation