Network Panel, The Scraper's Most Important Tool

If Elements is your reconnaissance kit, Network is the gold-mine. Most modern pages don't render data into HTML server-side, they ship a near-empty HTML shell, then fetch data via XHR / Fetch / WebSocket from JSON endpoints. Find those endpoints and you skip HTML parsing entirely. This is the foundational skill of Sub-Path 3.

Layout

• Top bar: Recording toggle (red dot), clear, search across all requests, preserve log toggle, disable cache toggle.
• Filter row: All / Doc / XHR / Fetch / JS / CSS / Img / Media / WS / Other / 3rd-party.
• Request list: name, status, type, initiator, size, time, waterfall.
• Detail pane: Headers / Preview / Response / Initiator / Timing for the selected request.

The five filter buttons that matter

Filter	What it shows	When you use it
Doc	Top-level HTML responses (page loads + redirects)	When mapping initial response
Fetch/XHR	Background data fetches (modern JS)	This is where 90% of your scraping wins live
JS	Script files	When reverse-engineering minified code (Sub-Path 3)
Img	Images	When checking image URLs / lazy-loading patterns
WS	WebSocket frames	When the data streams in real-time

You'll spend most of your Network time on Fetch/XHR. That's where the JSON endpoints live.

The workflow: "find the data endpoint"

A canonical recipe, works for almost any modern site:

1. Open DevTools → Network → Fetch/XHR filter.
2. Reload the page (Cmd/Ctrl + R).
3. Sort by Size descending. The big JSON responses are usually what you want.
4. Click a likely-looking response. Look at Preview, if it's the data you want, you've found the endpoint.
5. Right-click → Copy → Copy as cURL (covered in lesson F12).

In ten seconds you have a curl command that fetches the exact data, in JSON form. No HTML parsing. No browser automation. Just a clean API call.

The detail tabs

Click any request:

• Headers, request + response headers, status code, URL. The request URL (with query params expanded) is what you'll reproduce.
• Preview, rendered view. JSON gets a collapsible tree; HTML gets a thumbnail; images preview.
• Response, raw response body. Useful for grep-style debugging.
• Initiator, who triggered this request (which JS file, which line). Click through to Sources to see the code that made the call.
• Timing, TTFB, download time, DNS, TCP, SSL. Useful when comparing your scraper to the browser.

Filtering and searching

The search box (top of Network) searches across request bodies, response bodies, headers, URLs, all of it. Type a string you saw on the rendered page (a product name, an exact price) and DevTools tells you exactly which request returned that data.

This is the single most useful trick: search for a literal value from the page, and Network points you at the source.

Other filters in the Filter input:

• domain:api.example.com
• method:POST
• status-code:200
• larger-than:10k
• has-response-header:x-rate-limit

Combine them. method:POST status-code:200 domain:api.example.com is a one-line filter for "successful POST calls to the API."

The waterfall, what's slow, what's slowest

The right column shows a Gantt chart of when each request happens relative to page load. Useful for two things in scraping:

1. Ordering. What runs first? The initial HTML doc, then often the auth/config calls, then the data calls. The data calls usually depend on a token from the earlier calls, replicate the order in your scraper.

2. Identifying lazy-loaded calls. Calls that fire only on scroll, or only on a click, will appear later in the waterfall. Catalog108's /challenges/dynamic/infinite-scroll/* pages are perfect for spotting this, scroll, watch new requests appear.

"Preserve log" and "Disable cache"

Two toggles you should turn on every time:

• Preserve log, keeps requests across navigations. Without it, requests are cleared on every page load and you lose context.
• Disable cache (only while DevTools is open), forces fresh requests. Your browser cache can hide network calls; turning this off ensures you see everything your scraper would have to make.

Block, throttle, replay

Lesser-known but invaluable:

• Right-click → Block request URL. Simulates that endpoint being unreachable. Useful for understanding which calls the page actually needs vs. nice-to-have.
• Throttling (top bar, "No throttling" dropdown). Simulate slow 3G or offline. Useful when debugging lazy-load behaviour and timing-dependent JS.
• Right-click → Replay XHR. Re-runs the request with the same parameters. Useful for testing whether a response is deterministic or session-dependent.

A real example

You want to scrape product reviews from practice.scrapingcentral.com/products/1-yellow-ceramic-mug. In Network:

1. Reload with Fetch/XHR filter.
2. Spot a call to /api/products/1/reviews?page=1 returning JSON.
3. Preview shows { "reviews": [...], "next_cursor": "..." }.
4. You now know: the page loads HTML, then fetches reviews via JSON. You can skip the HTML and hit /api/products/1/reviews?page=N directly with requests. Add ?page=2, ?page=3 for pagination.

Five minutes of Network work; saves an hour of HTML parsing.

Hands-on lab

On practice.scrapingcentral.com/products/1-yellow-ceramic-mug (any product detail), find the XHR call that loads reviews. Note the URL, method, status, and what the response looks like. Then click "Copy as cURL" and paste it into your terminal, you should get the same JSON. Congratulations: you just bypassed HTML parsing for the reviews scrape.