Finding by ID, Class and Attribute

Most real scraping targets are styled or identified by attributes: an id for unique elements, a class for repeated elements, and data-* attributes added by the framework. find() and find_all() let you filter on any of them.

By ID

from bs4 import BeautifulSoup

html = "<div id="main"><p>content</p></div>"
soup = BeautifulSoup(html, "html.parser")
print(soup.find(id="main").get_text())  # content

By Class

Use the class_ keyword argument (trailing underscore) because class is reserved in Python:

html = "<ul><li class="item">A</li><li class="item promo">B</li></ul>"
soup = BeautifulSoup(html, "html.parser")

print(soup.find_all("li", class_="item"))  # both li's (first class matches)

# match a specific class in a multi-class attribute:
print(soup.find("li", class_="promo"))     # <li class="item promo">B</li>

By Arbitrary Attributes

Pass any attribute as a keyword; for attribute names that are not valid Python identifiers (dashes, dots), use the attrs dict:

html = "<a data-page="2" rel="next">next</a>"
soup = BeautifulSoup(html, "html.parser")

print(soup.find("a", rel="next"))
print(soup.find(attrs={"data-page": "2"}))
print(soup.find("a", data_page="2"))  # underscores map to data-page

Regex and Function Values

Attribute values can be regular expressions or functions:

import re
html = "<a href="https://x.com/a/1">1</a><a href="https://x.com/a/22">22</a>"
soup = BeautifulSoup(html, "html.parser")

print(soup.find_all("a", href=re.compile(r"/a/\d+$")))  # both

def deep_link(tag):
    return tag.has_attr("href") and tag["href"].startswith("https://x.com/a/2")
print(soup.find(deep_link))

Summary

Filter by id for unique elements, by class_ (with the underscore) for repeated classes, and by arbitrary attributes directly or through the attrs dict. Attribute values may be exact strings, regexes, or functions, which covers virtually any selection need a static page can present.