Reverse Engineering Hidden APIs

Almost every modern website fetches data from internal APIs that are not publicly documented. Finding and replicating those calls lets you extract structured data without parsing HTML.

The Process

1. Open DevTools, press F12 in Chrome, go to the Network tab
2. Filter by Fetch/XHR, this hides images, CSS, and scripts
3. Interact with the site, search, scroll, click "Load More"
4. Inspect interesting requests, look for JSON responses
5. Replicate in Python, copy URL, headers, and body

Extracting a Hidden API Call

After spotting a request in DevTools, right-click it and select "Copy as cURL." Then convert to Python:

import requests

# Discovered from network tab on an e-commerce site
url = "https://www.example-store.com/api/search"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept": "application/json",
    "X-Requested-With": "XMLHttpRequest",
    "Referer": "https://www.example-store.com/search?q=laptop",
}
params = {
    "q": "laptop",
    "page": 1,
    "per_page": 24,
    "sort": "relevance",
}

response = requests.get(url, headers=headers, params=params, timeout=15)
response.raise_for_status()

data = response.json()
for product in data["results"]:
    print(f"{product['name']} - ${product['price']}")

Common Patterns to Look For

Pattern	Where to Find It
Search autocomplete	Type in a search box, watch XHR requests
Infinite scroll	Scroll down, look for paginated API calls
Filter/sort	Apply filters, see what parameters change
Login/auth	Submit a form, capture the auth token response
Price/stock data	Open a product page, watch for JSON responses

Handling Anti-Scraping Headers

Hidden APIs often check for specific headers. Missing them returns 403 or empty data:

import requests

# These headers are commonly required
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.9",
    "Referer": "https://www.target-site.com/",
    "Origin": "https://www.target-site.com",
    "X-Requested-With": "XMLHttpRequest",
}

session = requests.Session()
session.headers.update(headers)

# First visit the main page to get cookies
session.get("https://www.target-site.com/", timeout=15)

# Now the API call will have the right cookies and headers
response = session.get(
    "https://www.target-site.com/api/v2/products",
    params={"category": "electronics"},
    timeout=15,
)
print(response.json())

Tips for Success

• Copy the exact headers from DevTools, omitting one can break things
• Check for CSRF tokens, some APIs require a token from a prior page load
• Watch for signed parameters, timestamps or hashes that change per request
• Test incrementally, start with all headers, then remove non-essential ones

When hidden APIs are behind heavy bot protection, ScraperAPI handles the fingerprinting, cookies, and IP rotation, so you can focus on the data.

Next Steps

• Use browser DevTools more effectively for API discovery
• Handle APIs that require session cookies
• Work with encrypted or obfuscated API parameters