Infinite Scroll, Five Implementation Patterns

"Infinite scroll" is an umbrella term for at least five distinct implementations. Each has a different trigger, a different reliable way to drive it, and almost always a different underlying API that you can hit directly. Knowing the pattern collapses two hours of debugging into ten lines.

The five patterns

Pattern	Trigger	Common API shape
IntersectionObserver	A sentinel element scrolls into view	?cursor=... or ?page=N
Scroll-event listener	window.scrollY exceeds threshold	?offset=N
Button + JS append	User clicks "Load more"	?page=N then DOM append
Cursor-paginated API (HTTP-only)	No scroll trigger, server returns next cursor	?cursor=...
Offset-paginated (HTTP-only)	Same, no scroll trigger needed	?offset=N&limit=20

The last two aren't really "infinite scroll" in the JS sense, they're paginated APIs hidden behind a scroll UI. From a scraper's perspective, the distinction is crucial: patterns 1–3 require scroll-triggering, patterns 4–5 don't.

Catalog108 ships the three JS-driven variants at /challenges/dynamic/infinite-scroll/intersection, /scroll-event, and /button-jsappend. Patterns 4 and 5 appear in the static-pagination labs from Sub-Path 1.

Pattern 1: IntersectionObserver

Modern, efficient, the dominant pattern in 2025+. A small "sentinel" element sits at the bottom of the list. When it enters the viewport, the observer fires a callback that fetches more.

How to detect it: in DevTools, find a hidden <div> near the end of the list, often empty or with a tiny height. Set a breakpoint on it. Scroll. The breakpoint fires before any data appears.

Driving it with Playwright:

page.goto("https://practice.scrapingcentral.com/challenges/dynamic/infinite-scroll/intersection")
page.wait_for_selector(".product-card")

while True:
  cards_before = page.locator(".product-card").count()
  page.locator(".product-card").last.scroll_into_view_if_needed()
  page.wait_for_function(
  "(n) => document.querySelectorAll('.product-card').length > n",
  arg=cards_before,
  timeout=5000,
  )
  if page.locator(".product-card").count() == cards_before:
  break

The trick: scroll the last card into view (which moves the sentinel below it into the viewport), then wait for the count to grow. If the count doesn't grow after a scroll, you've hit the end.

Pattern 2: scroll-event listener

Older pattern. Page listens to scroll events on window or a container, checks scrollTop + clientHeight >= scrollHeight - threshold, and fires a fetch when triggered.

How to detect it: in DevTools, attach an event listener breakpoint on scroll. Scroll. If the breakpoint hits an app handler (not the framework), it's this pattern.

Driving it:

page.goto("https://practice.scrapingcentral.com/challenges/dynamic/infinite-scroll/scroll-event")

prev = 0
for _ in range(50):  # safety cap
  page.mouse.wheel(0, 2000)
  page.wait_for_timeout(300)  # acceptable here, scroll-event uses debouncing
  curr = page.locator(".product-card").count()
  if curr == prev:
  break
  prev = curr

Two compromises here:

• page.mouse.wheel simulates real scrolling. Scroll-into-view can work but some implementations only react to wheel events.
• The 300ms wait is the one place a small wait_for_timeout is forgivable, scroll-event handlers are usually debounced (~150ms), so a short pause is genuinely required. Better: wait for the count to grow.

Pattern 3: button + JS append

A "Load more" button. Click, page fetches next batch, appends to DOM. The cleanest pattern.

page.goto("https://practice.scrapingcentral.com/challenges/dynamic/infinite-scroll/button-jsappend")

while True:
  cards_before = page.locator(".product-card").count()
  btn = page.locator("button.load-more")
  if btn.count() == 0 or not btn.is_visible():
  break
  btn.click()
  page.wait_for_function("(n) => document.querySelectorAll('.product-card').length > n",
  arg=cards_before, timeout=5000)

Loop until the button disappears or stops being visible. The post-click wait synchronises on the count.

Pattern 4: cursor-paginated API (HTTP)

Open DevTools Network. Scroll. Watch for an XHR like /api/products?cursor=eyJpZCI6MjB9. The response usually contains a next_cursor field. Hit the API directly:

import requests

cursor = None
all_products = []
while True:
  params = {"cursor": cursor} if cursor else {}
  r = requests.get("https://practice.scrapingcentral.com/api/products", params=params)
  data = r.json()
  all_products.extend(data["products"])
  cursor = data.get("next_cursor")
  if not cursor:
  break

Five lines of Python, no browser, no scrolling. This is the right strategy whenever the API is reproducible.

Pattern 5: offset-paginated (HTTP)

Same as #4 but with ?offset=N&limit=20 instead of cursors. Same code, simpler counter:

offset = 0
while True:
  r = requests.get("https://practice.scrapingcentral.com/api/products",
  params={"offset": offset, "limit": 20})
  page_items = r.json().get("products", [])
  if not page_items:
  break
  all_products.extend(page_items)
  offset += 20

The difference between cursor and offset matters: cursors are stable as new items are inserted; offsets shift. For ephemeral feeds (Twitter, Reddit), offsets give you duplicates and gaps.

How to choose the technique

Step 1: Open DevTools → Network → Fetch/XHR.
Step 2: Scroll once.
Step 3:
  - Did an XHR fire that returns the data?
  YES → use Pattern 4 or 5 (HTTP-only). Skip browser entirely.
  NO  → it's probably WebSocket or in-memory generation; need Pattern 1/2/3.
Step 4: If browser is required, which trigger?
  - IntersectionObserver: scroll the last card into view.
  - Scroll-event: simulate wheel scrolls.
  - Button: click the button.

In practice, ~70% of infinite-scroll pages have a discoverable API (patterns 4 or 5). Use it. The remaining 30% genuinely need a browser-driven trigger.

Anti-detection note

Some sites detect "instant scroll to the bottom" patterns and stop loading. If you find data stops streaming after a few aggressive scrolls:

• Slow scrolling (page.mouse.wheel(0, 500) instead of 5000).
• Random pauses between scrolls.
• Scroll relative to viewport position (window.scrollBy(0, window.innerHeight)).

These are anti-bot evasion tactics covered in Sub-Path 5. For now: stay realistic.

When the load stops without an end

Some sites paginate forever (the feed never ends). Your scraper needs an exit condition:

• A maximum scroll count (for _ in range(50)).
• A maximum item count (if len(items) > 500: break).
• A timestamp cut-off (if item["created_at"] < cutoff: break).

Without one, the loop runs until the browser tab dies of memory exhaustion.

Hands-on lab

Open /challenges/dynamic/infinite-scroll/intersection, then /scroll-event, then /button-jsappend. For each, write the corresponding driver loop and confirm you collect every product. Then open DevTools → Network on each, at least one of these pages calls an API you could hit directly with requests. That's your bonus exercise: find which one and bypass the browser entirely.