Scraping Links, Lists and Paragraphs

Three element families cover most "extract the content" tasks: anchor tags for URLs, list items for collections, and paragraphs for body text. The pattern is the same for all: select the elements, read the attributes or text, and normalize the result.

All Links

from bs4 import BeautifulSoup

html = '<a href="/home">Home</a><a href="https://x.com/a">A</a><a>no link</a>'
soup = BeautifulSoup(html, "html.parser")

for a in soup.find_all("a"):
    href = a.get("href")
    if href:
        print(href, "-", a.get_text(strip=True))

Turn relative URLs into absolute ones with urllib.parse.urljoin:

from urllib.parse import urljoin

base = "https://example.com/products/"
for a in soup.find_all("a"):
    href = a.get("href")
    if href:
        print(urljoin(base, href))

Lists and List Items

html = "<ul><li>alpha</li><li>beta</li></ul><ol><li>first</li></ol>"
soup = BeautifulSoup(html, "html.parser")

print([li.get_text(strip=True) for li in soup.find_all("li")])
# ['alpha', 'beta', 'first']

# only items in the first list:
print([li.get_text(strip=True) for li in soup.ul.find_all("li")])

Paragraphs and Headings

html = "<h2>Title</h2><p>Body one.</p><p>Body two.</p>"
soup = BeautifulSoup(html, "html.parser")

print(soup.h2.get_text(strip=True))
print([p.get_text(" ", strip=True) for p in soup.find_all("p")])

Structured Output

Collect the extracted values into lists of dicts (or rows) so they map cleanly onto CSV or database tables:

items = []
for li in soup.find_all("li", class_="product"):
    items.append({
        "name": li.select_one("h3").get_text(strip=True),
        "price": li.select_one("span.price").get_text(strip=True),
    })
print(items)

Summary

Scrape links with find_all("a") plus .get("href") and urljoin for absolute URLs, scrape lists by iterating li elements, and scrape prose with p and heading get_text() calls. Normalize every value (strip whitespace, build absolute URLs) and store structured rows, not raw strings.