Benchmarking SIMD vs Scalar Wasm Kernels

This guide measures what vectorising a kernel actually bought, in a way that survives scrutiny: two builds from one source, the same input, warmup discarded, runs interleaved, and results reported as a distribution rather than a single number.

SIMD speedup claims are unusually easy to overstate. The two builds differ in one flag, so it is tempting to time each once and divide. That number is almost always wrong — sometimes flatteringly, because the scalar run happened during tier-up, and sometimes pessimistically, because the vector run hit a thermal dip. The structure below removes both.

Interleave, do not run in blocks Running all scalar iterations and then all vector iterations lets a machine that warms up or throttles over time bias one variant. Alternating them means any drift affects both equally, so the comparison stays valid even on an imperfect machine. blocked — drift biases whichever ran later 200 × scalar 200 × SIMD clock drops as the machine heats interleaved — both see the same conditions …and so on The ratio of the two medians is then a property of the code rather than of when each variant happened to run.

Prerequisites

  • [ ] Two builds from the same source: one with +simd128, one without
  • [ ] A single input buffer both will read, generated once with a fixed seed
  • [ ] A browser you will report results from, and its version number
  • [ ] Nothing else running — other tabs and background builds move the numbers more than SIMD does

Step-by-step procedure

1. Instantiate both, outside the timed region

const [scalar, simd] = await Promise.all([
  WebAssembly.instantiateStreaming(fetch("/wasm/kernel.wasm")),
  WebAssembly.instantiateStreaming(fetch("/wasm/kernel.simd.wasm")),
]);

2. Build the input once and copy it into both memories

const N = 1 << 20;                       // 1M floats
const input = new Float32Array(N);
let seed = 42;
for (let i = 0; i < N; i++) {
  seed = (seed * 1664525 + 1013904223) >>> 0;   // deterministic, not Math.random
  input[i] = (seed >>> 8) / 0xffffff;
}

function load(inst) {
  const ptr = inst.instance.exports.alloc(N * 4);
  new Float32Array(inst.instance.exports.memory.buffer, ptr, N).set(input);
  return ptr;
}
const scalarPtr = load(scalar);
const simdPtr = load(simd);

Identical bytes on both sides is the requirement people most often miss — a fresh random array per variant makes the comparison meaningless if the data affects branch behaviour at all.

3. Warm up until the timings flatten

function warm(inst, ptr) {
  for (let i = 0; i < 50; i++) inst.instance.exports.scale(ptr, N, 1.0001);
}
warm(scalar, scalarPtr);
warm(simd, simdPtr);

4. Time them interleaved

const samples = { scalar: [], simd: [] };
for (let i = 0; i < 200; i++) {
  let t = performance.now();
  scalar.instance.exports.scale(scalarPtr, N, 1.0001);
  samples.scalar.push(performance.now() - t);

  t = performance.now();
  simd.instance.exports.scale(simdPtr, N, 1.0001);
  samples.simd.push(performance.now() - t);
}

5. Report a distribution

function stats(a) {
  const s = [...a].sort((x, y) => x - y);
  const q = (p) => s[Math.floor(s.length * p)];
  return { median: q(0.5).toFixed(3), p95: q(0.95).toFixed(3) };
}
console.table({ scalar: stats(samples.scalar), simd: stats(samples.simd) });
console.log("speedup:", (stats(samples.scalar).median / stats(samples.simd).median).toFixed(2) + "×");

6. Repeat at several input sizes

One size answers one question. Sweeping across sizes tells you where the crossover is, which is the result that actually informs a decision.

Expected output

┌─────────┬──────────┬─────────┐
│ (index) │  median  │   p95   │
├─────────┼──────────┼─────────┤
│ scalar  │ '2.184'  │ '2.611' │
│ simd    │ '0.831'  │ '1.104' │
└─────────┴──────────┴─────────┘
speedup: 2.63×

# sweeping N
N = 1024      scalar 0.004  simd 0.004   1.00×
N = 65536     scalar 0.141  simd 0.071   1.99×
N = 1048576   scalar 2.184  simd 0.831   2.63×
N = 16777216  scalar 41.20  simd 22.61   1.82×

The sweep is more informative than any single row. At a thousand elements the two are identical because the loop overhead dominates. At a million the vector version is at its best. At sixteen million the ratio falls again, because the kernel has become memory-bound and no amount of arithmetic throughput helps when the bottleneck is fetching the data.

The speedup curve has a peak At small sizes fixed overhead dominates and the two variants are indistinguishable. In the middle range the vector body dominates and the advantage is largest. At very large sizes the kernel becomes memory-bound and the advantage shrinks again. speedup input size → 1× — no difference overhead dominates compute-bound — the vector body pays fully memory-bound — loads are the limit Quote the size alongside the ratio. A speedup without an input size is not a reproducible claim.

Gotchas

The first few samples are wild. Warmup was too short. Print the first twenty timings once and see where they flatten; a fixed count copied from an example may not match your kernel.

The scalar build is unexpectedly fast. Check that it really is scalar — RUSTFLAGS set globally in .cargo/config.toml will vectorise both builds and produce a suspiciously equal comparison.

Results change between page loads. Something outside the benchmark is varying: another tab, a background download, or the browser’s own tier-up policy. Interleaving handles slow drift; for the rest, run more iterations and report the median.

The two builds produce different numbers. Expected for reductions and a bug for element-wise work. Assert equality on element-wise kernels and a tolerance on reductions, and treat a mismatch in the first category as a correctness failure that invalidates the timing entirely.

Timing includes the copy into linear memory. Then you are benchmarking set(), not the kernel. Copy once, outside the loop.

The checklist behind a defensible number Both variants must be built from the same source, read identical input bytes, be warmed until timings flatten, and be run interleaved. Any one of these missing invalidates the ratio, usually in the flattering direction. same source one flag apart identical input seeded, generated once warmed up until timings flatten interleaved not run in blocks Miss the first and you are comparing two different programs. Miss the second and the data drives the difference. Miss the third and you measured the baseline tier. Miss the fourth and you measured the machine warming up. All four failures tend to exaggerate the speedup, which is why published SIMD numbers deserve scepticism.

Performance note

The measurement itself costs something: performance.now() is clamped to a coarse resolution in most browsers to mitigate timing attacks, which means a single iteration of a fast kernel may be below the clock’s granularity. If per-iteration times are quantised — lots of identical values — batch several calls inside one timed region and divide, rather than trusting a resolution the browser does not provide.

Finally, remember what this number is for. A 2.6× kernel speedup translates into a whole-application improvement only in proportion to the time that kernel represented. Profiling the application first, then vectorising the function that actually dominates, is the order that produces user-visible results — the same discipline the benchmarking area applies to Wasm versus JavaScript comparisons.

Turning the measurement into a regression test

A one-off benchmark answers a question; a benchmark in CI keeps the answer true. The version that works in a pipeline asserts a ratio rather than an absolute time, because absolute times vary between runners while the ratio between two variants on the same machine is comparatively stable:

const scalarMedian = stats(samples.scalar).median;
const simdMedian = stats(samples.simd).median;
const speedup = scalarMedian / simdMedian;

console.log(`speedup ${speedup.toFixed(2)}×`);
if (speedup < 1.8) {
  throw new Error(`SIMD speedup regressed to ${speedup.toFixed(2)}× (expected ≥ 1.8×)`);
}

Pick the threshold well below the observed value — if the kernel measures 2.6×, assert 1.8× — so ordinary runner variance does not produce false failures. What this catches is the real regression: someone edits the kernel in a way that silently blocks vectorisation, and the ratio collapses toward 1.0 while both variants still produce correct results. Without the assertion that change ships unnoticed and the SIMD build quietly stops being worth its extra artifact.

Pair it with the correctness comparison in the same job. A speedup assertion alone can be satisfied by a kernel that got faster by being wrong:

assert.deepStrictEqual(readResult(scalar), readResult(simd), "variants diverged");

Reporting numbers other people can use

If the result is going into documentation or a decision, record the context alongside it. A speedup figure without the input size, the engine and the machine class is not reproducible, and the two most common ways published SIMD numbers mislead are omitting the size — where the crossover means small inputs show no gain — and omitting the browser, whose optimising tiers differ enough to move a ratio by tens of per cent.

A compact format that carries everything needed:

kernel: rgba→grayscale
input:  1 048 576 pixels
engine: Chrome 129 / V8 12.9, macOS, M2
result: scalar 2.184 ms (p95 2.611) · simd 0.831 ms (p95 1.104) · 2.63×

Five lines, and anyone can tell whether the figure applies to their situation. That is the difference between a benchmark and an anecdote.

Frequently Asked Questions

Should I benchmark in Node or in a browser? Both, for different purposes. Node is convenient and stable for regression tracking; the browser is where your users are, and its tier-up behaviour differs. Report browser numbers when making a claim.

How many iterations are enough? Enough that the median stops moving between runs — usually a few hundred for a kernel in the millisecond range. If the median shifts by more than a few per cent between two consecutive runs, you need more iterations or a quieter machine.

Can I compare against JavaScript at the same time? Yes, and it is worth doing — the interesting comparison is often scalar Wasm versus JavaScript versus SIMD Wasm. Use the same interleaving and the same input, and see measuring Wasm vs JavaScript throughput.

Should I benchmark the whole application or just the kernel? Both, in that order. The kernel measurement tells you whether the vectorisation worked; the application measurement tells you whether it mattered. A kernel that is three times faster and two per cent of the frame changes nothing a user can perceive, and knowing that before shipping a second artifact saves the complexity.

How do I compare across browsers fairly? Run the same page and the same input in each, and report each browser separately rather than averaging. Optimising tiers differ enough that a ratio from one engine does not transfer to another, and an average across engines describes no real user.

Is performance.now() precise enough? For a kernel measured in milliseconds, yes. For one measured in microseconds the clock’s deliberate coarseness dominates, and the fix is to time a batch of iterations together and divide.

← Back to Wasm SIMD & Vectorized Computation