How Web Scraping Works

Web scraping is the process of downloading web pages and extracting the data you care from the HTML. A scraper is just a loop of three steps: request, parse, extract.

The HTTP Cycle

1. Your client sends an HTTP request (usually GET) with headers like User-Agent.
2. The server responds with a status code and a body, typically HTML or JSON.
3. You parse the HTML into a tree and navigate it to pull out values.
4. You store the values (CSV, database, JSON) and repeat for the next page.

requests plus Beautiful Soup

The standard pairing is the requests library for fetching and Beautiful Soup for parsing:

import requests
from bs4 import BeautifulSoup

resp = requests.get(
    "https://example.com/products",
    headers={"User-Agent": "Mozilla/5.0 (research-bot)"},
    timeout=10,
)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")

for card in soup.select("div.product"):
    print(card.h2.get_text(strip=True))
    print(card.select_one("span.price").get_text(strip=True))

Pagination

Most list pages have a "next" link or a query parameter. Detect the next URL and loop with a delay between requests:

import time

url = "https://example.com/products?page=1"
while url:
    soup = BeautifulSoup(requests.get(url, timeout=10).text, "html.parser")
    # ... extract rows ...
    url = soup.select_one("a.next").get("href")
    time.sleep(1)  # be polite

Politeness and Robustness

• Set a descriptive User-Agent and respect robots.txt where it matters.
• Always use timeouts; raise_for_status() to catch HTTP errors early.
• Check for None before reading attributes: pages change structure without notice.
• Store raw HTML occasionally so you can re-parse without re-fetching.

Summary

Web scraping is a request-parse-extract loop. Fetch pages with requests using a User-Agent and timeouts, parse the response with Beautiful Soup, extract values with find or select, handle pagination in a loop, and store the results. Be polite: use delays, and design your scraper to survive layout changes.