Async HTTP with aiohttp

HTTP is the canonical I/O-bound workload, and aiohttp is the de-facto async HTTP client for Python. A single ClientSession manages the connection pool and TLS state; requests are coroutines, so a hundred fetches overlap on one thread. This chapter covers the client; the same session model underlies the aiohttp server you meet in web frameworks.

Single Request

import asyncio
import aiohttp

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get("https://api.github.com") as resp:
            print(resp.status)
            data = await resp.json()
            print(data)

asyncio.run(main())

The session is a context manager: entering it opens the connector, exiting it closes all pooled connections. The response is also async - await resp.json(), await resp.text(), or await resp.read() consume the body.

Fan-Out with gather

import asyncio
import aiohttp

URLS = [f"https://api.github.com/repos/python/cpython/tags?page={i}"
        for i in range(1, 6)]

async def fetch(session, url):
    async with session.get(url) as resp:
        return url, resp.status, len(await resp.read())

async def main():
    async with aiohttp.ClientSession() as session:
        results = await asyncio.gather(*(fetch(session, u) for u in URLS))
    for url, status, size in results:
        print(url[-12:], status, size)

asyncio.run(main())

All five requests are in flight simultaneously; total latency is roughly the slowest one, not the sum. For rate-limited APIs, wrap each call in a semaphore (as in the previous chapter) and set connector=aiohttp.TCPConnector(limit=10) to bound open connections.

Errors and Timeouts

import asyncio
import aiohttp

TIMEOUT = aiohttp.ClientTimeout(total=10)

async def safe_get(session, url):
    try:
        async with session.get(url, timeout=TIMEOUT) as resp:
            resp.raise_for_status()
            return await resp.json()
    except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
        print("failed", url, exc)
        return None

Always set a total timeout - an async request without one can suspend a task forever on a dead connection. Catch ClientError (connection failures, DNS) and TimeoutError separately from HTTPError, which raise_for_status() raises for 4xx/5xx responses you may want to retry.

Servers

The same library provides an async server: define a handler coroutine per route and run it with web.run_app. In practice most applications use a framework (FastAPI, Starlette) on top of the same async model, so the client skills here transfer directly to reading and writing async web APIs.

Summary

Use one ClientSession per program (or per worker), fan out requests with gather, cap concurrency with a connector limit and a semaphore, and give every request a total timeout. That combination is the standard recipe for fast, polite async scraping and API aggregation.