Headless vs Headed Browser Scraping

When using browser automation for scraping, you choose between two modes: headless (no visible window) and headed (visible browser window). This choice affects performance, resource usage, debugging ease, and detection risk.

What Is Headless Mode?

In headless mode, the browser runs without a graphical user interface. There is no visible window. The browser renders pages, executes JavaScript, and processes CSS in memory, but nothing is displayed on screen.

# Playwright, headless (default)
browser = p.chromium.launch(headless=True)

# Selenium, headless
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(options=options)

// Puppeteer, headless
const browser = await puppeteer.launch({ headless: 'new' });

What Is Headed Mode?

In headed mode, the browser opens a visible window. You can watch the scraper navigate, click, and scroll in real time.

# Playwright, headed
browser = p.chromium.launch(headless=False)

# With slow motion for debugging
browser = p.chromium.launch(headless=False, slow_mo=500)

Comparison

Factor	Headless	Headed
Speed	Faster (no rendering to screen)	Slower
Memory	Lower	Higher
Debugging	Harder (need screenshots/logs)	Easier (visual feedback)
Server compatible	Yes (no display needed)	Requires display or Xvfb
Bot detection	More easily detected	Less likely flagged
CI/CD friendly	Yes	No (without virtual display)

Detection Differences

Headless browsers have subtle differences that anti-bot systems detect. For example, Chrome headless historically had:

• Different navigator.plugins array (empty)
• Different User-Agent string containing "Headless"
• Different WebGL renderer strings
• Missing Chrome-specific JavaScript APIs

Modern versions of Chrome have reduced these differences, but some detection services still distinguish between modes.

# Check if running headless (anti-bot test)
is_headless = page.evaluate("""
    () => {
        return navigator.plugins.length === 0
            || navigator.userAgent.includes('Headless');
    }
""")

New Headless Mode in Chrome

Chrome 112+ introduced a new headless mode that is virtually identical to headed mode:

# Playwright uses the new headless by default
browser = p.chromium.launch(headless=True)

# Selenium, new headless
options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

Running Headed on a Server

If you need headed mode on a Linux server (for anti-detection), use a virtual display:

sudo apt-get install xvfb
xvfb-run python scraper.py

Or use pyvirtualdisplay in Python:

from pyvirtualdisplay import Display

display = Display(visible=0, size=(1920, 1080))
display.start()

# Now launch your browser in headed mode
browser = p.chromium.launch(headless=False)

Recommendation

Use headless mode for production scraping. Use headed mode only for development and debugging. If bot detection is a concern, apply stealth techniques or use a managed service like ScrapingAnt which handles browser fingerprinting and detection evasion on your behalf.

Next Steps

• Learn to intercept network requests with Playwright
• Explore browser fingerprinting and stealth mode
• Set up parallel browser scraping