Skillversitas

Error Handling and Best Practices

Production scrapers fail in predictable ways: pages change, elements go missing, the server errors, and encodings get weird. Defensive code, timeouts, and a small set of conventions keep a scraper from crashing or silently producing garbage.

Guard Every Lookup

find() returns None, and attribute access on a missing tag raises. The habit to build: check before you use.

price_tag = soup.find("span", class_="price")
price = price_tag.get_text(strip=True) if price_tag else None

# missing attributes return None from .get, not a KeyError:
href = link.get("href")          # None if absent
href = link["href"] if link.has_attr("href") else None

Try/except Around Fragile Steps

import requests
from bs4 import BeautifulSoup

try:
    resp = requests.get(url, timeout=10, headers={"User-Agent": UA})
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "lxml")
except (requests.RequestException, ValueError) as e:
    log.warning("fetch failed for %s: %s", url, e)
    soup = None

Working Conventions

  • Store raw HTML next to your extracted data so you can re-parse after a site redesign.
  • Use a consistent selector strategy (CSS selectors for structure, find for logic) and centralize selectors as constants so layout changes are one-line fixes.
  • Log every row that came out with missing fields instead of crashing the run.
  • Respect robots.txt and add a delay (time.sleep or a rate limiter) between requests.

Debugging Techniques

print(soup.prettify())                    # see what the parser built
print(type(soup.find("div")))             # Tag or None?
print(soup.find("div").contents if soup.find("div") else "none")

When a selector stops working, dump the relevant subtree and compare it to what you expect; the cause is usually a class-name change or a new wrapper div.

Summary

Make scrapers defensive: guard find() results against None, use .get for attributes, wrap fetching in try/except with timeouts and raise_for_status, centralize selectors, log incomplete rows, store raw HTML, and be polite with robots.txt and delays. These habits turn a one-off script into a scraper that survives real websites.