Remote Data with httpfs

The httpfs extension lets DuckDB read and write data directly from HTTP(S) and S3 endpoints without downloading files first. Parquet files on a web server, an S3 bucket, or a CDN can be queried in place.

Loading the Extension

In current DuckDB versions httpfs is bundled and auto-loads the first time you reference an http or s3 path, but you can also load it explicitly:

INSTALL httpfs;
LOAD httpfs;
SELECT * FROM read_parquet('https://duckdb.org/data/1.parquet') LIMIT 3;

Reading from S3

Public buckets need no credentials at all. For private buckets or S3-compatible stores, set the credentials and endpoint before the query:

SET s3_region='auto';
SET s3_access_key_id='AKIA...';
SET s3_secret_access_key='...';
SET s3_endpoint='minio.local';   -- for MinIO and S3-compatible stores
SELECT * FROM 's3://my-bucket/events/*.json' LIMIT 5;

Glob patterns work in the path, so you can scan an entire directory or partition prefix in one query.

Writing to Remote Storage

COPY TO also works over HTTP(S) and S3. Use an explicit destination file name:

COPY (SELECT * FROM range(100))
TO 'https://bucket.endpoint/data/out.parquet' (FORMAT PARQUET);

Practical Tips

• Use glob patterns on the source side: *.parquet, part-*.csv.
• Remote Parquet queries are fast because DuckDB reads only the needed column chunks with predicate pushdown.
• For slow networks, raise the timeout: SET httpfs_timeout=30000;

Summary

With the httpfs extension DuckDB queries remote Parquet, CSV, JSON, and Arrow files directly over HTTP, HTTPS, or S3. Load the extension, configure S3 credentials when needed, and reference remote paths in FROM or read_* functions. COPY TO writes results back to the same remote locations.