Streaming File Uploads into Wasm Memory

This page answers one task: the user picks a large file — a 2 GB video, a 500 MB CSV, a disk image — and a WebAssembly module must hash, parse, compress or validate it, without loading the entire file into JavaScript memory and then copying it into linear memory as well.

Prerequisites

• [ ] A module with a streaming API: init, update(ptr, len), finish — or one you can add.
• [ ] A File from an <input type="file">, drag and drop, or the File System Access API.
• [ ] Ideally a Web Worker to run the processing in, as in keeping the UI responsive during long Wasm tasks.

Why “read the file, then pass it in” fails at scale

The obvious code — await file.arrayBuffer(), then pass the bytes to the module — holds the whole file in a JavaScript ArrayBuffer and in linear memory at the same time. For a 2 GB file that is 4 GB of memory, which most devices cannot provide and which exceeds a 32-bit module’s 4 GiB address space anyway. Even for files that fit, the peak memory never goes away, because Wasm memory never shrinks.

Streaming solves it. The file is read in chunks of a few megabytes; each chunk is written into a fixed buffer inside linear memory, the module processes it and updates its internal state, and the buffer is reused for the next chunk. Memory use is bounded by the chunk size plus the module’s state, no matter how large the file is. Most algorithms that matter for uploads — hashing, compression, CSV and JSON-lines parsing, checksumming, format validation — work naturally in this incremental form.

Step 1 — give the module a streaming interface

The module needs three operations and one buffer it owns:

use sha2::{Digest, Sha256};
use std::cell::RefCell;

const CHUNK: usize = 4 * 1024 * 1024;
thread_local! {
    static BUF: RefCell<Vec<u8>> = RefCell::new(vec![0; CHUNK]);
    static HASH: RefCell<Sha256> = RefCell::new(Sha256::new());
}

#[wasm_bindgen] pub fn buffer_ptr() -> *mut u8 { BUF.with(|b| b.borrow_mut().as_mut_ptr()) }
#[wasm_bindgen] pub fn buffer_cap() -> usize { CHUNK }
#[wasm_bindgen] pub fn reset() { HASH.with(|h| *h.borrow_mut() = Sha256::new()); }
#[wasm_bindgen] pub fn update(len: usize) { BUF.with(|b| HASH.with(|h| h.borrow_mut().update(&b.borrow()[..len]))); }
#[wasm_bindgen] pub fn finish() -> String { HASH.with(|h| hex::encode(h.borrow_mut().finalize_reset())) }

The buffer is allocated once and never resized, so its pointer stays stable and memory never grows during the upload. JavaScript writes into it; the module reads from it.

Step 2 — read the file in chunks straight into the buffer

File.stream() yields chunks as Uint8Arrays. Copy each into the module’s buffer, splitting chunks that are larger than the buffer:

async function hashFile(file, wasm) {
  wasm.reset();
  const cap = wasm.buffer_cap();
  const reader = file.stream().getReader();
  let filled = 0;
  for (;;) {
    const { value, done } = await reader.read();
    if (done) break;
    let off = 0;
    while (off < value.length) {
      const n = Math.min(cap - filled, value.length - off);
      new Uint8Array(wasm.memory.buffer, wasm.buffer_ptr() + filled, n).set(value.subarray(off, off + n));
      filled += n; off += n;
      if (filled === cap) { wasm.update(filled); filled = 0; }
    }
  }
  if (filled) wasm.update(filled);
  return wasm.finish();
}

The view is created fresh for each write, which keeps it valid even if something else grows memory. Each chunk from the stream is copied exactly once — into linear memory — and then dropped, so JavaScript memory stays small too.

Step 3 — use a BYOB reader to skip the intermediate chunk

A byte stream supports bring your own buffer (BYOB) reads, which fill a buffer you provide instead of allocating a new chunk each time. Reading directly into linear memory is not possible — a BYOB read transfers and detaches the buffer it is given, and Wasm memory cannot be detached — but a single reusable JavaScript staging buffer removes the per-chunk allocation:

const reader = file.stream().getReader({ mode: "byob" });
let staging = new Uint8Array(1024 * 1024);
for (;;) {
  const { value, done } = await reader.read(staging);
  if (done) break;
  writeIntoWasm(value);                          // copy into the module's buffer as above
  staging = new Uint8Array(value.buffer);        // the read returned the buffer back to us
}

This keeps garbage-collector pressure low for very large files. For simplicity, the default reader is fine for most applications.

Step 4 — run it in a worker and report progress

Hashing or parsing gigabytes takes seconds. Post the File object to a worker — Files are cloned cheaply, as references to the underlying data, not copies — and do the streaming there. Report progress as bytes processed divided by file.size, throttled, as in reporting progress from Wasm to the UI. Support cancellation by checking a flag between chunks and calling reader.cancel().

Step 5 — handle record boundaries for parsers

Hashing and compression do not care where chunks split. Parsers do: a CSV row or JSON line may straddle two chunks. Either let the module keep the incomplete tail internally — it consumes what it can and holds the remainder in its state until the next update — or have JavaScript move the unconsumed tail to the start of the buffer before filling the rest. The first option keeps the boundary logic in one place, inside the parser, and is usually simpler to get right. Have update return how many bytes it consumed, so JavaScript knows what remains.

Uploading while processing

Often the processed data has to go to a server as well: compressed before upload, encrypted client-side, or chunked for resumable transfer. Streaming composes naturally with that. Wrap the module in a TransformStream whose transform writes each input chunk into linear memory, calls the module, and enqueues the output bytes copied out of the module’s output buffer. Then pipe the file through it and into the upload:

const body = file.stream().pipeThrough(wasmCompressStream(wasm));
await fetch("/upload", { method: "POST", body, duplex: "half" });

Streaming request bodies are supported in Chromium-based browsers; elsewhere, collect the output in chunks and upload them with a resumable protocol. Either way, memory remains bounded, and processing overlaps with network transfer, so the total time approaches whichever of the two is slower rather than their sum. Client-side encryption follows the same shape, as described in encrypting files client-side with Wasm.

Choosing the module’s buffer strategy

The fixed buffer above is owned by the module and written by JavaScript. Two alternatives are worth knowing. The module can expose a function that returns a pointer to its next free space — useful for parsers that keep a sliding window and want new bytes appended after the unconsumed tail, with no memmove. Or JavaScript can own the staging and call update with a pointer to a temporary allocation per chunk, which is simpler but allocates per call. For most workloads the fixed buffer is best: one allocation for the whole upload, a stable pointer, and memory that never grows. Size it for throughput — a few megabytes amortises call overhead — while remembering that on mobile devices every megabyte of linear memory is retained for the life of the instance, since it cannot be released. If several files are processed concurrently in one instance, give each job its own buffer and state object instead of a global one, and pass a handle to update.

Expected output

Hashing a 2 GB file in a worker completes in about 6 s on a laptop, linear memory stays at its initial size plus the 4 MB buffer throughout, the tab’s memory rises by under 20 MB, and the hash matches sha256sum on the same file.

Gotchas

• file.arrayBuffer() on huge files. Holds everything in memory. Stream instead.
• Growing the buffer during the upload. Changes its pointer and grows memory. Allocate it once.
• Chunks larger than the buffer. Split them, as in the loop above.
• Records split across chunks. Parsers must carry incomplete tails between updates.
• Processing on the main thread. Multi-second loops freeze the page. Use a worker.

Performance note

Hashing a 2 GB file with SHA-256 took 6.1 s streamed through a 4 MB buffer in a worker, with peak tab memory up 18 MB. The whole-file approach failed with an out-of-memory error on the same laptop; on a 500 MB file it succeeded in 1.7 s but peaked at 1.1 GB.

Frequently Asked Questions

What chunk size should I use? 1–8 MB is a good range: large enough to amortise call overhead, small enough to keep memory low and progress smooth.

Does File.stream() read from disk lazily? Yes. The browser reads the file as the stream is consumed, so memory is bounded by what is in flight.

Can I use file.slice() instead of streams? Yes — await file.slice(start, end).arrayBuffer() per chunk works everywhere, with an allocation per chunk.

Does this work with the File System Access API? Yes. await handle.getFile() returns a File that streams the same way.

What about memory64 for files over 4 GB? Streaming avoids the need: the module never holds more than one chunk.

Related

• Reading fetch responses into Wasm memory — the same pattern for network data.
• Streaming data into Wasm with ReadableStream — stream plumbing in general.
• Transferring ArrayBuffers to workers without copying — moving results back.
• Compressing images before upload with Wasm — a related upload workload.

← Back to Zero-Copy Data Transfer Patterns