Rotating User Agents for Web Scraping

Every HTTP request includes a User-Agent header that identifies the browser or tool making the request. Default Python libraries send obvious bot signatures like python-requests/2.31.0, which websites instantly flag.

Why Rotate User Agents?

Websites track User-Agent strings to detect scrapers. Sending hundreds of requests with the same UA, especially a non-browser one, is one of the easiest ways to get blocked.

Building a User Agent Rotator

import requests
import random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_4_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14.4; rv:125.0) Gecko/20100101 Firefox/125.0",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
]

def scrape_with_random_ua(url):
    headers = {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
    }
    response = requests.get(url, headers=headers, timeout=30)
    return response

# Each request uses a different User-Agent
for i in range(5):
    resp = scrape_with_random_ua("https://httpbin.org/headers")
    ua = resp.json()["headers"]["User-Agent"]
    print(f"Request {i+1}: {ua[:60]}...")

Using the fake-useragent Library

Instead of maintaining your own list, use the fake-useragent package which pulls from a real-world UA database:

from fake_useragent import UserAgent
import requests

ua = UserAgent()

for _ in range(5):
    headers = {"User-Agent": ua.random}
    resp = requests.get("https://httpbin.org/user-agent", headers=headers)
    print(resp.json())

Install it with pip install fake-useragent.

Best Practices

• Keep UAs up to date, outdated browser versions are a red flag
• Match UA to other headers, a Chrome UA with Firefox-specific headers looks suspicious
• Pair with proxy rotation, rotating UAs alone is not enough for aggressive anti-bot systems
• Use only desktop UAs unless you specifically need mobile pages

The Easy Route: ScraperAPI

Services like ScraperAPI handle User-Agent rotation automatically alongside proxy rotation and CAPTCHA solving:

import requests

API_KEY = "YOUR_SCRAPERAPI_KEY"
url = f"http://api.scraperapi.com?api_key={API_KEY}&url=https://example.com"
response = requests.get(url)

Next Steps

• Combine UA rotation with proxy rotation for stronger anti-detection
• Learn about browser fingerprinting beyond just User-Agent strings