Querying Parquet Files with DuckDB-Wasm

This guide answers one task: query a Parquet dataset that lives on a CDN directly from the browser, downloading only the parts a query needs, and get results back as something you can render without copying every value twice.

Prerequisites

  • [ ] @duckdb/duckdb-wasm from npm, plus its worker and .wasm bundles served from your origin.
  • [ ] Parquet files reachable over HTTPS with Accept-Ranges: bytes and permissive CORS.
  • [ ] A dataset worth the trouble — below a few megabytes, fetching the whole file is simpler.
  • [ ] Familiarity with columnar formats; this page assumes you know what a row group is.

Why this works at all

Parquet stores data column by column, in row groups, with a footer describing where everything lives and statistics for each chunk. Given range requests, an engine can read the footer, decide which row groups and which columns a query touches, and fetch only those byte ranges.

The consequence is striking in practice. A query selecting two columns with a date filter over a 2 GB dataset might download 10–30 MB: the footer, the two columns’ chunks for the matching row groups, and nothing else. The filter is applied twice — once coarsely using the min and max statistics to skip row groups entirely, then precisely on the rows that remain.

This only holds if the file is laid out for it. A Parquet file written as one enormous row group defeats the skipping, and a file sorted randomly with respect to your filter column means every row group matches the range and nothing is skipped. Writing the data sorted by the column you filter on is the single most effective thing you can do for query performance, and it costs nothing at query time.

What a filtered query actually downloads The footer is read first, its statistics eliminate row groups whose value ranges cannot match, and only the requested columns within the surviving row groups are fetched. Everything drawn without a border is never transferred. row group 1 — ts 2024-01 … 2024-03 ts amount payload (skipped) notes (skipped) row group 2 — ts 2024-04 … 2024-06 (eliminated by statistics) no bytes fetched for this row group at all footer schema · offsets · min/max per chunk read first, in one small range request Sorting the file by the column you filter on is what makes the elimination work; an unsorted file matches every row group and downloads everything.

Initialise the engine

DuckDB-Wasm ships several bundles — with and without threads, with and without exception handling — and picks the right one for the current browser. Serve them from your own origin so the selection logic can fetch them without cross-origin complications.

import * as duckdb from '@duckdb/duckdb-wasm';

const BUNDLES = {
  mvp:  { mainModule: '/vendor/duckdb/duckdb-mvp.wasm',  mainWorker: '/vendor/duckdb/duckdb-browser-mvp.worker.js' },
  eh:   { mainModule: '/vendor/duckdb/duckdb-eh.wasm',   mainWorker: '/vendor/duckdb/duckdb-browser-eh.worker.js' },
};

const bundle = await duckdb.selectBundle(BUNDLES);
const worker = new Worker(bundle.mainWorker);
const db = new duckdb.AsyncDuckDB(new duckdb.ConsoleLogger(), worker);
await db.instantiate(bundle.mainModule);
const conn = await db.connect();

The engine runs in the worker from the start, which means every query is asynchronous and the main thread is never blocked. That is the correct default for an analytical engine, where a single query can legitimately run for seconds.

Query a remote file directly

Point the engine at a URL and it handles the rest. No registration, no download step, no schema declaration — the footer tells it everything.

await conn.query(`INSTALL httpfs; LOAD httpfs;`);   // some builds have it already

const result = await conn.query(`
  SELECT date_trunc('day', ts) AS day, count(*) AS n, sum(amount) AS total
  FROM read_parquet('https://data.example.com/txns/2024.parquet')
  WHERE ts >= TIMESTAMP '2024-03-01' AND ts < TIMESTAMP '2024-04-01'
  GROUP BY 1 ORDER BY 1
`);

console.table(result.toArray().map((r) => r.toJSON()));

Several files are as easy as one — a glob or an array reads a partitioned dataset, and the engine parallelises across them:

SELECT region, sum(amount) AS total
FROM read_parquet(['https://data.example.com/txns/2024-0{1,2,3}.parquet'])
GROUP BY region ORDER BY total DESC;

Reading results without copying everything

Results come back as Arrow tables: columnar, typed, and backed by contiguous buffers. Converting them to JavaScript objects with toArray().map(r => r.toJSON()) is convenient and allocates an object per row — fine for a hundred rows, wasteful for a hundred thousand.

For anything large, read the columns directly. A numeric column comes out as a typed array you can pass straight to a chart or a compiled kernel without a per-row loop.

const table = await conn.query('SELECT day, total FROM summary ORDER BY day');
const days   = table.getChild('day').toArray();     // typed, contiguous
const totals = table.getChild('total').toArray();
renderChart(days, totals);                          // no per-row objects created

That distinction is the difference between a dashboard that updates in 10 ms and one that stutters. It is also why aggregating in SQL is doubly valuable: fewer rows cross the boundary and the ones that do arrive in a shape rendering code can use directly.

Registering local files

Users bring their own data too. A File from an input can be registered as a virtual file and queried with the same SQL, with no upload and no server involvement.

const file = input.files[0];
await db.registerFileHandle(file.name, file, duckdb.DuckDBDataProtocol.BROWSER_FILEREADER, true);
const preview = await conn.query(`SELECT * FROM read_parquet('${file.name}') LIMIT 100`);

CSV works the same way through read_csv_auto, which sniffs types and headers. For a data tool this is a remarkably short path from “user drops a 500 MB file” to “user runs SQL against it”, and nothing leaves the machine.

Three sources, one query language Remote Parquet over range requests, a local file the user picked, and tables created in memory all appear to SQL as relations. Queries can join across them without any explicit import step. remote Parquet range requests only user's local file never uploaded SQL engine in a worker joins across all sources columnar, vectorised Arrow result typed columns, no per-row cost Joining a user's local CSV against a reference table on your CDN is one query, and no data crosses the network in either direction except the ranges the query needs.

Expected output

A well-targeted query downloads a small fraction of the file. Confirm it in the network panel rather than trusting the design:

GET 2024.parquet  Range: bytes=2145612288-2145616383   4 KB    (footer)
GET 2024.parquet  Range: bytes=812339200-819113983     6.5 MB  (ts column, rg 3–5)
GET 2024.parquet  Range: bytes=1209384960-1223851519   13.8 MB (amount column, rg 3–5)
query: 1.9 s, 20.3 MB transferred of a 2.1 GB file

Seeing a single request for the entire file means range requests are not working — usually a server or CDN that does not honour Range, or a CORS configuration that hides Accept-Ranges from the browser.

Memory: the tab still has a budget

The engine keeps intermediate results in linear memory, and an unbounded aggregation will exhaust it just as it would on a server with too little RAM. Set a limit explicitly so failures are clean rather than a tab crash:

SET memory_limit = '512MB';
SET threads = 4;

Queries that exceed the limit fail with a clear out-of-memory error, which you can catch and turn into a message suggesting a narrower filter. That is a far better experience than the tab disappearing. Grouping by a high-cardinality column — a user identifier, a raw URL — is the usual way to blow the budget, and the usual fix is to aggregate at a coarser grain first.

How little of the file a query needs Columnar layout plus row-group statistics means a selective query reads a fraction of the file. Downloading the whole thing throws that advantage away. download whole file 480 MB transferred before the query starts range requests, all columns 148 MB row groups pruned by statistics range requests, two columns 27 MB only the columns the query names are read at all The server must support range requests and CORS, or every strategy collapses back to a full download. Naming columns explicitly instead of selecting everything is the single largest saving available.

Gotchas

  • Failed to fetch on the Parquet URL. Missing CORS headers. The file must allow your origin and expose Accept-Ranges.
  • The whole file downloads. The server ignores Range, or the file has a single row group so nothing can be skipped. Check both.
  • Could not read Parquet footer. The URL returned HTML — usually a 404 page or a login redirect served with status 200.
  • Query is fast, rendering is slow. Results were converted to per-row objects. Read Arrow columns directly instead.
  • Out of memory on a large group-by. Set memory_limit, reduce cardinality, or pre-aggregate the dataset when you write it.
  • Timestamps are off by hours. Parquet stores timestamps with a unit and sometimes a zone; check what the writer produced before blaming the engine.

Performance note

Over a 2.1 GB dataset partitioned into monthly files sorted by timestamp, a one-month filtered aggregate transferred about 20 MB and completed in 1.9 s on a laptop over a 100 Mbit connection — most of it network. The same query against an unsorted single-row-group file transferred 2.1 GB and never finished in a tab. Layout, not engine tuning, is where the performance lives.

Frequently Asked Questions

How large a dataset is realistic? Tens of gigabytes remotely, provided queries are selective and the files are laid out well. What must fit in the tab is the working set — the columns and rows a query touches — not the dataset.

Can I join remote data against the user’s file? Yes, in one query. That is one of the more compelling reasons to use this engine in a browser at all.

Should I use this instead of SQLite? For analytics over columnar data, yes. For transactional application state with many small writes, no — use SQLite, and consider both in the same product for their respective jobs.

How should I partition the files I publish? By the column users filter on most, usually time, with each file covering a period that keeps individual files in the tens to low hundreds of megabytes. That gives the engine file-level pruning before row-group pruning even starts, and it makes incremental publishing trivial — a new month is a new file rather than a rewrite.

Does compression help or hurt here? Helps, substantially. Parquet compresses per column chunk, so a compressed file transfers less for the same query and decompresses inside the worker at gigabytes per second. Zstandard is the usual best tradeoff; leaving the data uncompressed to “save CPU” almost always costs more in transfer than it saves.

← Back to Databases & Persistent Storage in Wasm