SEO Monitoring and SERP Scraping Guide

Tracking your search rankings is essential for SEO. While tools like Ahrefs and SEMrush offer this, scraping SERPs directly gives you more control and flexibility.

What SERP Data to Track

• Organic rankings, Your position for target keywords
• Featured snippets, Whether you own position zero
• People Also Ask, Related questions Google shows
• Local pack, Google Maps results in search
• Ads, Who is bidding on your keywords
• Rich results, Reviews, FAQs, sitelinks

Scraping Google Search Results

Google is aggressive about blocking scrapers. Using ScraperAPI is the most reliable approach.

import requests
from bs4 import BeautifulSoup

API_KEY = "YOUR_SCRAPERAPI_KEY"

def scrape_serp(keyword, location="us"):
    url = f"https://www.google.com/search?q={keyword}&gl={location}&num=20"
    
    resp = requests.get(
        f"http://api.scraperapi.com?api_key={API_KEY}&url={url}&autoparse=true"
    )
    return resp.json()  # ScraperAPI returns structured SERP data

ScraperAPI Auto-Parse for Google

ScraperAPI offers structured SERP data without manual HTML parsing:

results = scrape_serp("best web scraping tools")

for result in results.get("organic_results", []):
    print(f"Position: {result['position']}")
    print(f"Title: {result['title']}")
    print(f"URL: {result['link']}")
    print(f"Snippet: {result['snippet']}")
    print("---")

Building a Rank Tracker

import json
from datetime import date

def track_rankings(keywords, domain):
    rankings = {}
    
    for keyword in keywords:
        serp = scrape_serp(keyword)
        position = None
        
        for result in serp.get("organic_results", []):
            if domain in result.get("link", ""):
                position = result["position"]
                break
        
        rankings[keyword] = {
            "position": position,
            "date": str(date.today()),
        }
    
    with open("rankings.jsonl", "a") as f:
        f.write(json.dumps(rankings) + "\n")
    
    return rankings

Monitoring Competitors in SERPs

Track which competitors appear for your target keywords:

Keyword	Your Position	Competitor A	Competitor B
"web scraping api"	3	1	5
"scraping proxy"	7	2	4
"data extraction"	12	8	3

SERP Feature Tracking

Beyond organic rankings, monitor SERP features:

• Featured snippets, Track which queries show snippets and who owns them
• People Also Ask, Find content ideas from related questions
• Knowledge panels, Monitor brand search results
• Image packs, Track visual search opportunities

Best Practices

1. Use ScraperAPI with auto-parse, Structured SERP data without HTML parsing
2. Track consistently, Same time, same location, same parameters
3. Monitor mobile vs desktop separately, Rankings differ
4. Use geographic targeting, Rankings vary by location
5. Track weekly, not daily, Rankings fluctuate naturally
6. Use ScrapingAnt as backup, Diversify your SERP scraping sources
7. Combine with Search Console data, Scraped SERPs + GSC gives the full picture