Reading Parquet Files

Parquet is a columnar storage format built for analytics, and it is the format DuckDB is designed to handle best. Because Parquet stores data column by column, with statistics stored per row group, DuckDB can read only the columns and row groups a query actually needs. This keeps large analytical scans fast and memory friendly, and it means you never have to import a file into a table before querying it.

Querying Parquet with read_parquet()

The read_parquet() table function queries a .parquet file directly, as if it were a table:

SELECT * FROM read_parquet('sales.parquet') LIMIT 10;
-- equivalent shorthand notation
SELECT * FROM 'sales.parquet' LIMIT 10;

DuckDB reads the file footer, discovers the schema, and applies column projection automatically: if you select two columns out of forty, only those two columns are read from disk.

Filter Pushdown and Row Groups

A Parquet file is divided into row groups, and each column chunk within a row group stores minimum and maximum statistics. DuckDB uses these statistics to skip row groups that cannot contain matching rows:

SELECT sum(amount) AS total
FROM read_parquet('sales.parquet')
WHERE order_date >= DATE '2024-01-01';

Useful reader options include file_row_number=true, which adds a file_row_number column with the row's position inside its file, and binary_as_string=true, which forces binary columns to be read as strings instead of BLOB.

Reading Multiple Files with Globs

Glob patterns query many files at once. All matched files must have compatible schemas:

-- all files in one directory
SELECT count(*) FROM read_parquet('data/sales-*.parquet');
-- recursive search over a directory tree
SELECT count(*) FROM read_parquet('data/**/*.parquet');

When files are organized in Hive-style folders such as year=2024/month=3/part.parquet, pass hive_partitioning=true. DuckDB parses the key=value directory names into virtual year and month columns, and pruning skips folders whose values fail the query's filters.

Inspecting the Schema

Three helpers reveal what is inside a file without reading all of its data:

DESCRIBE SELECT * FROM 'sales.parquet';
SELECT * FROM parquet_schema('sales.parquet');
SELECT row_group_id, column_name, row_group_num_rows,
       compression
FROM parquet_metadata('sales.parquet') LIMIT 5;

DESCRIBE SELECT lists the logical columns and types, parquet_schema() lists the physical Parquet columns, and parquet_metadata() exposes per-row-group details such as compression and row counts.

Writing Parquet with COPY

DuckDB also writes Parquet through the COPY statement, producing either one file or a partitioned directory tree:

COPY (SELECT * FROM orders) TO 'orders.parquet' (FORMAT PARQUET);
COPY orders TO 'out_dir'
    (FORMAT PARQUET, PARTITION_BY (year, region));

A partitioned output directory can be read back with read_parquet('out_dir/**/*.parquet', hive_partitioning=true), which makes COPY ... TO and read_parquet() a natural round trip for ETL jobs.

Summary

DuckDB treats Parquet as a native data source. read_parquet() and the quoted file path notation query files directly, glob patterns and hive_partitioning cover file collections, DESCRIBE, parquet_schema() and parquet_metadata() expose structure, statistics-driven filter pushdown skips irrelevant row groups, and COPY ... TO writes single or partitioned Parquet output. Together these features make DuckDB an efficient engine for both ad hoc inspection and full pipelines over Parquet datasets.