Network Tab Deep Dive, Every Filter and Why

DevTools' Network panel is where 90% of API-scraping discovery happens. Most beginners use 10% of it and miss the request they need. Senior scrapers know every column, filter, and right-click action.

This lesson is a guided tour.

Open the panel cleanly

In Chrome / Edge:

• F12 or Cmd+Option+I → Network tab.
• Preserve log, check it. Without it, the log clears on navigation, and a redirect-driven login flow will wipe your evidence.
• Disable cache, check it. Otherwise a 304 won't show response bodies.
• Filter type: "All" by default. Switch to "Fetch/XHR" as your first move on most targets.

You should see this in the toolbar:

[●] [⊘] [☑ Preserve log] [☑ Disable cache] [Throttling: No throttling]
Filter: [____________]  [Invert]  [Hide data URLs]
[All] [Fetch/XHR] [JS] [CSS] [Img] [Media] [Font] [Doc] [WS] [Wasm] [Manifest] [Other]

Filter types, what each means

• Fetch/XHR, JSON API calls. Your primary target. Catches fetch(), XMLHttpRequest, and most GraphQL.
• Doc, top-level HTML documents (navigations, iframes). Useful when login flow involves a form POST that returns HTML.
• JS, script files. Use when hunting for apiKey, signRequest, or other functions in bundles.
• CSS / Img / Media / Font, almost always noise for scrapers; mute them.
• WS, WebSocket connections. Click one and the panel switches to a frame view (lesson 3.43).
• Wasm, WebAssembly modules. Rarely relevant unless reverse-engineering anti-bot.
• Manifest, PWA manifests. Almost never relevant.

For most scraping sessions: Fetch/XHR first, then JS if you need to find where a value is computed.

Text filter, small box, huge power

The filter input supports more than substring matching:

• mime-type:application/json, only JSON responses.
• status-code:200 or status-code:401, specific HTTP status.
• method:POST, only POSTs.
• domain:api.example.com, only that domain.
• larger-than:10k, only responses bigger than 10 KB.
• has-response-header:x-csrf-token, only requests whose response carries that header.
• is:running or is:from-cache, request state.
• Regex: wrap in slashes: /products\?page=\d+/

Combine them: method:POST mime-type:application/json status-code:200, only successful JSON POSTs. Surgical.

Columns, what to add

Right-click the column header → reveal the hidden ones. Scraper-relevant columns:

• Name, request path.
• Method, GET/POST/PUT.
• Status, 200/204/302/401/etc.
• Type, fetch / xhr / document / etc.
• Initiator, what JS line or call kicked it off. Critical for tracing where in the bundle a request comes from.
• Size, bandwidth (Transferred) vs payload (Content).
• Time, duration of the request.
• Waterfall, visual timing.

Useful additions: Cookies (Set-Cookie count), Set-Cookie (the actual values), Has overrides (if you're using DevTools overrides).

Remove Type and Time if they're noise to you. Keep Initiator always.

Per-request inspection

Click any request → side panel opens with sub-tabs:

• Headers, request and response headers. Where you'll spend most of your time.
• Payload, for POST/PUT, the body. Switch between "view URL-encoded" and "view source" to see what shape the body is.
• Preview, formatted JSON tree. Best when you want to glance at structure.
• Response, raw body. Copy this if Preview hides anything.
• Initiator, call stack of the JS that triggered the request. Click any frame to jump to Sources.
• Timing, DNS, TCP, TLS, request, response. Useful for debugging hanging requests.
• Cookies, what cookies were sent with this request and what cookies the response set.

The Initiator tab is the secret weapon. Click apiClient.js:42 and you're standing in the function that built the request, including the signing logic, header construction, and any token sources. Critical for the reverse-engineering lessons (3.20, 3.21, 3.45).

Right-click power moves

Right-click any request, and a dozen useful options appear:

• Copy → Copy as cURL, your bread and butter (lesson F12).
• Copy → Copy as fetch, paste into Console to re-run with one tweak.
• Copy → Copy as PowerShell / Node.js fetch, language variants.
• Copy → Copy response, just the body. Good for prototyping a parser without re-fetching.
• Copy → Copy URL with parameters, query string preserved.
• Block request URL / Block request domain, stub out an endpoint and watch the page fail; tells you which requests are load-bearing.
• Save all as HAR with content, export the whole session. See below.
• Replay XHR, re-fire the request right from DevTools. Great for tweaking params interactively (Console becomes a REPL).

HAR files, capture the whole session

A HAR is a JSON archive of every Network entry, request, response, timing, headers, body. Two uses:

1. Bug reports. Share a HAR with a teammate; they can replay it locally.
2. Offline analysis. Tools like Postman, Insomnia, and mitmweb can import HARs. Useful when you've captured something fragile (a fresh signed request) and want to study it without re-running.

Right-click anywhere in the Network log → "Save all as HAR with content." Strip cookies before sharing.

A discovery workflow

Step-by-step, against practice.scrapingcentral.com/products/1-white-wooden-vase:

1. Open DevTools → Network → Fetch/XHR. Preserve log + Disable cache on.
2. Hard-reload (Cmd+Shift+R).
3. Scan the request list. Two of them stand out: /api/products/1 and /api/products/1/reviews.
4. Click /api/products/1. Preview tab, you see clean JSON: {id, name, price, description, images...}. That's the main data source.
5. Headers tab, note any auth header. None here (public endpoint). Note User-Agent is browser-default; not required (verify by stripping in Python).
6. Right-click → Copy as cURL. Paste in terminal. Confirm 200 + same JSON.
7. Translate to requests.get(...). Done.

For comparison, run the same flow on /products (the list view), you'll see /api/products?page=1. Note how identical the pattern is: the site has a 1:1 mapping between routes and JSON endpoints.

Python: replaying a captured request

If you Copy as cURL and then run curlconverter (or pip's curl_cffi.curl_to_python), you'll get something like:

import requests

cookies = {"session": "abc123"}
headers = {
  "accept": "application/json",
  "user-agent": "Mozilla/5.0 ...",
}
r = requests.get(
  "https://practice.scrapingcentral.com/api/products/1",
  cookies=cookies,
  headers=headers,
)
r.json()

Strip what isn't needed (you'll learn the minimum-viable-request game in lesson 3.8).

Hands-on lab

Open practice.scrapingcentral.com/products/1-white-wooden-vase in your browser with Network → Fetch/XHR open. Inventory every JSON request. For each one, note: method, URL, status, and one interesting field from the response. Then practise three right-click actions: Copy as cURL, Block request URL (then reload, does the page break?), and Save all as HAR. By the end you should feel like you can take the Network panel apart and put it back together, it's the scraper's primary instrument.