HTML to Text Conversion

Converting an HTML fragment to clean plain text sounds trivial, but the details matter: scripts and styles must go, tags become nothing, whitespace must be collapsed, and structure (paragraphs, lists) should become line breaks where it matters.

Step 1: Remove Noise

from bs4 import BeautifulSoup

html = "<body><script>var x=1;</script><style>p{color:red}</style><p>Real text</p></body>"
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style"]):
    tag.decompose()
print(soup.get_text(" ", strip=True))  # Real text

Step 2: get_text with Control

html = "<div><p>Para one.</p><p>Para two.</p><ul><li>a</li><li>b</li></ul></div>"
soup = BeautifulSoup(html, "html.parser")

print(soup.get_text())                  # glued together
print(soup.get_text("\n", strip=True))  # paragraphs and items on lines
print(soup.get_text(" ", strip=True))   # single spaces

Better Line Breaks

For documents where block structure matters, add newlines per block element instead of a flat separator:

def to_text(soup):
    parts = []
    for el in soup.find_all(["p", "h1", "h2", "h3", "li", "blockquote"]):
        t = el.get_text(" ", strip=True)
        if t:
            parts.append(t)
    return "\n\n".join(parts)

print(to_text(soup))

Whitespace Handling

• strip=True trims each fragment; then collapse runs of spaces: " ".join(text.split()).
• Preserve intentional newlines inside pre tags by handling them before decompose.
• HTML entities (&, <) are decoded automatically by the parser; you see literal text, not markup.

Summary

HTML-to-text is a pipeline: decompose script and style tags, call get_text with a separator (newline for blocks, space for inline) and strip=True, and post-process whitespace. For documents with headings and lists, extract block elements individually and join with blank lines to preserve readability.