When Browser Automation Is the Right Tool (And When It Isn't)

The most useful lesson in this sub-path is "don't use a browser." That sounds strange in a sub-path about browser automation, but it's the truth: most pages that look dynamic can be scraped without one. This lesson is the decision framework.

The honest cost of a browser

For every page load, compare:

Metric	requests	Playwright (headless)
Time per page	50–200 ms	1–5 s
Memory per worker	~30 MB	~200–500 MB
Concurrency on 8 GB	100s of workers	~10 workers
Network bytes	HTML only	HTML + JS + CSS + images + fonts
Failure modes	TLS, timeout	TLS, timeout, JS errors, race conditions, browser crashes
Detection risk	Low	High (webdriver, headless flags)

A browser costs roughly 10–50× more in time, 10× more in memory, and adds an entire category of failures. If you're going to pay that price, the page had better need it.

Five scenarios where browser automation IS required

These are the genuine cases. Outside them, you should default to HTTP.

1. Pure CSR with no exposed API. The browser fetches data, but the data lives in WebSocket messages or in an internal-only endpoint with anti-replay tokens. You can't reproduce the call.
2. Heavy client-side computation. Charts, canvases, custom-rendered text, content that doesn't exist until JS draws it. (Sub-Path 2 lessons on canvas and shadow DOM.)
3. Sites with aggressive TLS or JS fingerprinting. Cloudflare's JS challenges, PerimeterX, DataDome, these gate every request behind a browser environment. (Sub-Path 5.)
4. Authentication flows that need real interaction. OAuth with mandatory user prompts, CAPTCHA-gated logins, SMS 2FA. You're driving a real session, not replaying tokens.
5. Interactive behaviour as the data. Hover-revealed tooltips, drag-and-drop ordering, scrolling-revealed content that the API doesn't expose paginated. The interaction is the scrape.

That's roughly it. Five buckets. Everything else has a faster path.

The five scenarios people THINK require a browser (and don't)

These come up constantly and waste hours. Run the three-test diagnostic from Lesson 2.1 before believing them:

1. "The data isn't in view-source." True for hybrid sites, but the data is right there in __NEXT_DATA__. Parse the JSON.
2. "There's an animated loading spinner." Cosmetic. The data is fetched via XHR, find the call.
3. "The page uses React/Vue." Framework choice doesn't determine rendering mode. Many React sites are SSR/SSG.
4. "It requires login." Sessions are cookies; cookies can be sent by requests. Lesson 1.4 and Sub-Path 4 cover this.
5. "There's pagination / infinite scroll." Almost always backed by a paginated API. Find the call and increment the page parameter.

In each case, ten minutes in DevTools usually beats two hours of Playwright debugging.

The checklist

Before you commit to a browser-based scrape, work through this list:

[ ] I ran View Source and the data isn't in the response HTML.
[ ] I ran 'Disable JavaScript' and the page failed to render the data.
[ ] I searched the response for hydration payloads
  (__NEXT_DATA__, __NUXT__, window.__INITIAL_STATE__) and found none.
[ ] I opened DevTools → Network → Fetch/XHR and either:
  (a) found no relevant call, OR
  (b) the call requires a dynamic token I cannot reproduce.
[ ] I confirmed the page works in a real browser (not blocked).
[ ] My target volume justifies the resource cost.

If you cannot tick all six, write the HTTP version first. You can always fall back to a browser later.

Hybrid strategies are common

Real-world scrapers often combine both:

• Browser to log in, HTTP to scrape. Drive Playwright once to handle a multi-step login or CAPTCHA, capture cookies, pass them to requests for the bulk crawl. Resource cost: one browser launch, then HTTP speed forever.
• Browser to discover, HTTP to harvest. Use Playwright on one page to watch the XHR calls, reverse-engineer them, then drop into HTTP for the production scrape. Lesson 2.23.
• Browser to bootstrap session, HTTP to continue. Anti-bot gates often only fire on the first request. After the JS challenge passes, the session cookie lets HTTP continue normally.

Pure-browser scrapers are slow. Pure-HTTP scrapers are sometimes impossible. Hybrid is usually the engineering sweet spot.

When you SHOULD reach for a browser early

There are cases where the calculation flips and starting with a browser saves time:

• One-shot or low-volume jobs. Scraping one page, once, manually-triggered. The 5-second browser overhead is irrelevant.
• Visual confirmation matters. You need a screenshot, a PDF, a visual diff.
• Complex multi-step interactions. Filling a 12-field form with conditional logic, where parameter discovery would take longer than the form itself.
• Prototyping. Get the data with a browser, see if the project is worth it, then optimise.

The mistake is using a browser at production scale when an HTTP scraper would have worked.

The mental decision tree

Start
  │
  ▼
Is the data in the HTML response?  ─ YES ─► HTTP. Done.
  │ NO
  ▼
Is it in a hydration payload?  ─ YES ─► HTTP + JSON parse. Done.
  │ NO
  ▼
Can I reproduce the XHR call?  ─ YES ─► HTTP to the API. Done.
  │ NO
  ▼
Is the page behind JS challenges?  ─ YES ─► Stealthed browser. (Sub-Path 5)
  │ NO
  ▼
Is the interaction itself the data?  ─ YES ─► Browser. Justified.
  │ NO
  ▼
Browser. But verify volume justifies cost.

Use this every time. The framework saves you from cargo-culting Playwright onto problems that don't need it.

Hands-on lab

This lesson has no lab page. Instead: pick three real sites you've considered scraping, news, e-commerce, a SaaS dashboard. For each, run the diagnostics from Lesson 2.1 and walk through the checklist above. Note which bucket each falls into. By the end you'll have a personal calibration of how often "needs a browser" is actually true. Spoiler: less often than you'd guess.