Browser Pool Patterns for Concurrency

Once your scraper outgrows "one page at a time", you face a memory-bounded resource problem. Spinning up a fresh browser per URL is too expensive. Sharing one page across many URLs serialises everything. The right answer is a bounded pool of contexts under a single browser process, fast, safe, and recoverable.

The shapes that fail

Anti-pattern 1: browser-per-URL.

for url in urls:
  with sync_playwright() as p:
  browser = p.chromium.launch()
  # ...
  browser.close()

Pays a 1-second launch cost per URL. For 1000 URLs that's 17 minutes of nothing but launch overhead.

Anti-pattern 2: one page, sequential.

browser = p.chromium.launch()
page = browser.new_page()
for url in urls:
  page.goto(url)
  # ...

Correct but slow. No parallelism. Throughput equals one-page-at-a-time.

Anti-pattern 3: unbounded parallelism.

await asyncio.gather(*[scrape(url) for url in urls])  # 1000 URLs in flight

Memory exhaustion. Each page is ~50 MB; 1000 pages is 50 GB. Process dies.

The right shape: bounded context pool

One browser. N contexts (or pages, see below). A queue of URLs. Workers pull URLs and process them through the pool.

import asyncio
from playwright.async_api import async_playwright

CONCURRENCY = 5

async def worker(name, queue, context, results):
  while True:
  try:
  url = queue.get_nowait()
  except asyncio.QueueEmpty:
  return
  page = await context.new_page()
  try:
  await page.goto(url, timeout=15000)
  results.append({
  "url": url,
  "title": await page.title(),
  "products": await page.locator(".product-card").count(),
  })
  except Exception as e:
  results.append({"url": url, "error": str(e)})
  finally:
  await page.close()

async def main(urls):
  queue = asyncio.Queue()
  for u in urls:
  queue.put_nowait(u)
  results = []

  async with async_playwright() as p:
  browser = await p.chromium.launch()
  context = await browser.new_context()

  workers = [worker(f"w{i}", queue, context, results) for i in range(CONCURRENCY)]
  await asyncio.gather(*workers)

  await browser.close()

  return results

One browser. One context (shared cookies, fine if all URLs are the same site). Five workers pulling from a queue. Each opens a page, scrapes, closes the page. Workers exit when the queue is empty.

Throughput: roughly CONCURRENCY × per-page-time. Five workers at 1.5 seconds per page = ~3 pages/second steady-state.

Per-URL contexts (when isolation matters)

If you need per-URL cookie isolation (multi-site crawl, multi-account scraping), spin up a context per URL:

async def worker(name, queue, browser, results):
  while True:
  try:
  url = queue.get_nowait()
  except asyncio.QueueEmpty:
  return
  context = await browser.new_context()
  try:
  page = await context.new_page()
  await page.goto(url)
  results.append({"url": url, "title": await page.title()})
  finally:
  await context.close()

Contexts are cheap (~10 ms), so creating one per URL is fine. The cost is mostly the goto, which dominates either way.

Per-worker contexts (the sweet spot)

For most production scrapers, the right pattern is one context per worker, reused for the worker's lifetime:

async def worker(name, queue, browser, results):
  context = await browser.new_context()
  try:
  while True:
  try:
  url = queue.get_nowait()
  except asyncio.QueueEmpty:
  return
  page = await context.new_page()
  try:
  await page.goto(url)
  results.append({"url": url, "title": await page.title()})
  finally:
  await page.close()
  finally:
  await context.close()

Worker keeps one context. Pages come and go inside it. Cookies accumulate per worker (sometimes you want this, sometimes not, it lets the worker maintain a session across URLs).

Right-sizing concurrency

Pick CONCURRENCY based on:

Constraint	Formula
Memory per page	~30-50 MB × concurrency
Available RAM	Less than 70% of total
Target rate limit	Less than the site's per-IP budget
Network bandwidth	Each page pulls 1-5 MB; saturating uplink helps no one

For an 8 GB laptop, 5-8 concurrent pages is typical. For a 32 GB server, 20-30 is fine. Beyond that, the target's rate limit (not your machine) becomes the bottleneck.

Crash recovery

Browsers crash. Playwright's browser may emit a disconnected event:

browser_lock = asyncio.Lock()
browser = await p.chromium.launch()
browser.on("disconnected", lambda: asyncio.create_task(restart_browser()))

async def restart_browser():
  nonlocal browser
  async with browser_lock:
  browser = await p.chromium.launch()

For most scrapers, simpler than crash-recovery is chunked scrapes with retries: process 50 URLs, save progress, start a new browser for the next 50. If one browser dies, only that chunk is lost.

Node Playwright equivalent

const { chromium } = require("playwright");
const pLimit = require("p-limit");

(async () => {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const limit = pLimit(5);

  const urls = [...]; // load your URL list
  const results = await Promise.all(urls.map(url => limit(async () => {
  const page = await context.newPage();
  try {
  await page.goto(url, { timeout: 15000 });
  return { url, title: await page.title() };
  } catch (e) {
  return { url, error: String(e) };
  } finally {
  await page.close();
  }
  })));

  await browser.close();
  console.log(results);
})();

p-limit(5) is the Node equivalent of the asyncio.Queue + worker pattern. Higher-level libraries (p-queue, bull, bullmq) add retry, priority, and persistence.

Long-running pools

For scrapers that run for hours, periodic browser restarts are healthy. Memory accumulates; long-lived browsers slow down. Restart every N tasks or every M minutes:

TASKS_PER_BROWSER = 200

async def chunked_main(urls):
  while urls:
  chunk = urls[:TASKS_PER_BROWSER]
  urls = urls[TASKS_PER_BROWSER:]
  async with async_playwright() as p:
  browser = await p.chromium.launch()
  # ... pool over chunk ...
  await browser.close()

Each chunk runs in a fresh browser. Memory stays bounded. Crashes only lose one chunk.

Resource cleanup at scale

Stale contexts, leaked pages, half-finished requests, these compound. Guardrails:

# Set a strict per-context timeout on actions
context = await browser.new_context()
context.set_default_timeout(10000)
context.set_default_navigation_timeout(20000)

# Track open pages
@contextmanager
async def tracked_page(context):
  page = await context.new_page()
  try:
  yield page
  finally:
  if not page.is_closed():
  await page.close()

Strict timeouts mean a stuck page fails after 10-20 seconds instead of holding a worker indefinitely. The tracked-page helper guarantees close() runs.

Hands-on lab

Open /products. Build a 5-worker async pool that scrapes all 10 paginated product list pages concurrently. Measure throughput. Then increase to 10 workers and measure again, note when you hit the target rate limit (the page starts returning 429 or hanging). That's the practical ceiling, not your machine's.