Using Browser DevTools to Find APIs

Chrome DevTools is your best friend for discovering the API calls a website makes behind the scenes. Every piece of dynamic content on a page, search results, product listings, user profiles, is typically fetched from an API endpoint.

Opening DevTools

1. Press F12 (or Cmd+Option+I on Mac) to open DevTools
2. Click the Network tab
3. Check Preserve log to keep requests across page navigations
4. Select the Fetch/XHR filter to show only API calls

Step-by-Step: Finding an API

Interact with the site while watching the Network tab. For example, searching on a site:

1. Type a search query and press Enter
2. Look for new XHR requests in the Network panel
3. Click on a request to inspect it
4. Check the Preview tab, if it shows JSON, you found an API
5. Switch to the Headers tab to see the full URL, method, and headers

Copying a Request to Python

Right-click any request in DevTools and choose Copy > Copy as cURL (bash). Then convert it:

# Converted from a cURL export
import requests

url = "https://www.example.com/api/search"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept": "application/json",
    "Cookie": "session_id=abc123; preferences=lang_en",
}
params = {"q": "web scraping", "page": 1}

response = requests.get(url, headers=headers, params=params, timeout=15)
data = response.json()

for result in data.get("items", []):
    print(result["title"])

You can also use the curlconverter package to automate the conversion:

pip install curlconverter

import curlconverter

curl_command = 'curl "https://api.example.com/data" -H "Authorization: Bearer token"'
python_code = curlconverter.to_python(curl_command)
print(python_code)

Useful DevTools Features

Feature	How to Access	Use Case
Filter by URL	Type in the filter box	Find specific endpoints
Replay request	Right-click > Replay XHR	Test if the endpoint works
Block request	Right-click > Block URL	See what data disappears from the page
Search all requests	Cmd+F in Network tab	Find requests containing specific text
Preserve log	Checkbox at top	Keep requests across page navigations

Identifying Required Headers

Strip headers one at a time to find the minimum set. Start with all headers from DevTools, then remove them to see which ones are actually needed. Common essential headers:

• User-Agent, almost always required
• Referer, many APIs check this
• Authorization or Cookie, for authenticated endpoints
• X-Requested-With, some APIs look for this

For sites with JavaScript challenges that block direct API access, ScrapingAnt renders pages in a real browser and can return the API responses your script needs.

Next Steps

• Reverse engineer complex hidden APIs
• Handle APIs requiring cookies and sessions
• Use Postman for deeper API exploration