CSS Selectors for Web Scraping

CSS selectors let you target specific HTML elements with precision. BeautifulSoup's select() method accepts CSS selector syntax.

Basic Selectors

from bs4 import BeautifulSoup

html = """
<div class="product" id="item-1">
  <h2 class="title">Widget Pro</h2>
  <span class="price" data-currency="USD">$29.99</span>
  <a href="/products/widget-pro">Details</a>
</div>
"""

soup = BeautifulSoup(html, "html.parser")

# By tag
soup.select("h2")              # All h2 elements

# By class
soup.select(".price")          # Elements with class "price"

# By ID
soup.select("#item-1")         # Element with id "item-1"

# By attribute
soup.select("[data-currency]") # Elements with data-currency attribute
soup.select('a[href^="/products"]')  # Links starting with /products

Combining Selectors

# Descendant (space)
soup.select("div.product h2")     # h2 inside div.product

# Direct child (>)
soup.select("div.product > span") # span directly inside div.product

# Multiple selectors (comma)
soup.select("h2, .price")         # All h2 AND all .price elements

Selector Cheat Sheet

Selector	Meaning
tag	Element by tag name
.class	Element by class
#id	Element by ID
[attr]	Element with attribute
[attr=val]	Attribute equals value
[attr^=val]	Attribute starts with value
[attr$=val]	Attribute ends with value
[attr*=val]	Attribute contains value
parent child	Descendant selector
parent > child	Direct child selector

Pro Tip

Use your browser's DevTools (F12) to test CSS selectors. Right-click an element, select "Inspect", and use the Console to run document.querySelectorAll(".your-selector") to verify your selector matches the right elements before writing Python code.