Retry, Backoff, and Rate Limit Handling

A scraper that retries naively is worse than one that doesn't retry at all. Hammering a rate-limited endpoint with no backoff gets your IP banned within minutes. A scraper that retries the right things, the right way, runs for months.

This lesson is about that "right way."

What's worth retrying

Three classes of errors:

1. Transient, retry with backoff.

• 429 Too Many Requests
• 500, 502, 503, 504 (server-side hiccups)
• Connection errors, timeouts, DNS failures
• Empty / truncated responses on a normally-JSON endpoint

2. Auth, refresh and retry once.

• 401 Unauthorized, token expired; refresh and try again
• 403 Forbidden, sometimes auth, sometimes anti-bot; usually retry once after re-auth, then give up

3. Permanent, do not retry.

• 400 Bad Request, your request is malformed; retrying won't fix it
• 404 Not Found, the resource doesn't exist
• 422 Unprocessable Entity, validation failed
• Anything 4xx that isn't 401, 403, 408, 429

Exponential backoff with jitter

The classic recipe:

• Attempt 1: try immediately.
• Attempt 2: sleep base seconds.
• Attempt 3: sleep base * 2.
• Attempt 4: sleep base * 4.
• ... up to base * 2^(N-1).

With jitter to avoid thundering herds (multiple workers retrying in lockstep):

import random, time

def sleep_backoff(attempt: int, base: float = 1.0, cap: float = 60.0) -> None:
  delay = min(cap, base * (2 ** attempt))
  # full jitter, pick uniformly in [0, delay]
  time.sleep(random.uniform(0, delay))

For 1s base, the schedule (with jitter) becomes roughly:

• Retry 1: 0–1s
• Retry 2: 0–2s
• Retry 3: 0–4s
• Retry 4: 0–8s
• Retry 5: 0–16s

Capped so you never wait an hour. Five attempts, ~30s worst case.

Respecting Retry-After

A 429 or 503 response often carries a Retry-After header:

• Retry-After: 30, wait 30 seconds.
• Retry-After: Wed, 21 Oct 2025 07:28:00 GMT, wait until that timestamp.

Always honor it:

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(header_value: str) -> float:
  if header_value.isdigit():
  return float(header_value)
  try:
  when = parsedate_to_datetime(header_value)
  return max(0, (when - datetime.now(timezone.utc)).total_seconds())
  except Exception:
  return 0

A complete retry decorator

import requests, time, random, logging
from functools import wraps

log = logging.getLogger(__name__)

class PermanentError(Exception): pass

def with_retry(max_attempts: int = 5, base: float = 1.0, cap: float = 60.0):
  def decorator(fn):
  @wraps(fn)
  def wrapper(*args, **kwargs):
  last_exc = None
  for attempt in range(max_attempts):
  try:
  return fn(*args, **kwargs)
  except requests.HTTPError as e:
  status = e.response.status_code
  if status in (400, 404, 422):
  raise PermanentError(f"{status} {e}") from e
  if status == 429 or 500 <= status < 600:
  # Honor Retry-After if present
  ra = e.response.headers.get("Retry-After")
  if ra:
  delay = retry_after_seconds(ra)
  else:
  delay = min(cap, base * (2 ** attempt))
  delay = random.uniform(0, delay)
  log.warning(f"{status} on attempt {attempt+1}; sleeping {delay:.1f}s")
  time.sleep(delay)
  last_exc = e
  continue
  raise
  except (requests.ConnectionError, requests.Timeout) as e:
  delay = min(cap, base * (2 ** attempt))
  delay = random.uniform(0, delay)
  log.warning(f"{type(e).__name__} on attempt {attempt+1}; sleeping {delay:.1f}s")
  time.sleep(delay)
  last_exc = e
  continue
  raise last_exc
  return wrapper
  return decorator

Usage:

@with_retry(max_attempts=5)
def fetch_page(page: int):
  r = requests.get(
  "https://practice.scrapingcentral.com/challenges/api/rest/rate-limited",
  params={"page": page},
  timeout=10,
  )
  r.raise_for_status()
  return r.json()

for p in range(1, 11):
  data = fetch_page(p)
  print(p, len(data.get("items", [])))

The decorator is reusable across any function returning a requests response.

Using built-in retry: HTTPAdapter + Retry

urllib3 ships an Retry class that mounts onto a requests.Session. Cleaner for simple cases:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
  total=5,
  backoff_factor=1.0,
  status_forcelist=[429, 500, 502, 503, 504],
  allowed_methods=["GET", "POST"],
  respect_retry_after_header=True,
)
adapter = HTTPAdapter(max_retries=retry)

s = requests.Session()
s.mount("https://", adapter)
s.mount("http://", adapter)

Trade-off: built-in is simpler but less customizable. Use it when you don't need bespoke logic; use the decorator when you do (e.g. auth refresh on 401, conditional retry on response body content).

PHP: Guzzle retry middleware

use GuzzleHttp\Client;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

$retryDecider = function (
  int $retries,
  RequestInterface $request,
  ?ResponseInterface $response = null,
  ?\Throwable $exception = null
): bool {
  if ($retries >= 5) return false;
  if ($exception) return true; // connection errors
  if ($response) {
  $status = $response->getStatusCode();
  if ($status === 429 || $status >= 500) return true;
  }
  return false;
};

$retryDelay = function (int $retries, ResponseInterface $response = null): int {
  if ($response && $response->hasHeader('Retry-After')) {
  $ra = $response->getHeaderLine('Retry-After');
  return is_numeric($ra) ? (int)$ra * 1000 : 1000;
  }
  return (int)(min(60, pow(2, $retries)) * 1000 * (mt_rand(0, 1000) / 1000));
};

$stack = HandlerStack::create();
$stack->push(Middleware::retry($retryDecider, $retryDelay));

$client = new Client([
  'base_uri' => 'https://practice.scrapingcentral.com',
  'handler'  => $stack,
  'timeout'  => 10,
]);

Same logic, different syntax.

Two anti-patterns

1. Retrying 400/404/422. They're permanent. Retrying multiplies the load on the server (and your logs) for no possible benefit.
2. Constant-interval retry. time.sleep(5) between every attempt floods servers during outages. Always use exponential backoff with jitter.

Hands-on lab

Catalog108's /challenges/api/rest/rate-limited endpoint enforces a tight per-IP limit (e.g. 5 requests / 10 seconds). Build a scraper that hits it 20 times. Without retries: most will 429. With the with_retry decorator above: all 20 succeed, with the scraper backing off when limited. Watch the Retry-After header in DevTools and confirm your scraper honors it.