Why CS Fundamentals Matter for Scrapers

This sub-path is optional. If you've finished Foundations and Static Scraping and your scrapers work, you can ship things. So why bother with data structures, algorithms, and Big-O?

Because at four specific moments in a scraping career, the lack of CS fundamentals turns into a wall:

1. Your scraper slows to a crawl past a few hundred thousand URLs and you don't know why.
2. A simple dedup step eats all your RAM and you're running it on a Pi.
3. You sit down for a job interview and the first whiteboard question is "implement a URL crawler with BFS."
4. You read a real crawler's source (Scrapy, Colly, Heritrix) and the data structures it uses look unfamiliar.

This sub-path covers the minimum to push past all four. It's not a CS degree, it's the CS shaped like the work you're actually doing.

The four areas, and which scraping problem each one solves

Area	Scraping problem it solves	Lesson
Data structures	"How do I dedup 100M URLs without dying?", "How do I order which page to fetch next?"	CS2
Algorithms & traversal	"How does a crawler decide what to visit next?", "How do I search 10M scraped products fast?"	CS3
Big-O complexity	"Why did my scraper start crawling slower as it ran longer?", "Why does memory grow without limit?"	CS4
Computability	Mostly nothing. Shows up only as interview trivia.	Briefly in CS4

Two of these (data structures, Big-O) you'll feel every day. One (algorithms) shows up the moment you write your own crawler. The fourth is the lightest-weight, P vs NP is a fun thing to know about but it won't change your code.

The red-flag moments, in detail

1. The scraper that mysteriously slows down

You wrote a scraper that visits 100 URLs in 10 seconds. You let it run on 1,000,000 URLs and it takes a week. You expected ~30 hours.

Common cause: you kept a visited = [] list and checked if url not in visited for every new URL. That's O(n) per check, multiplied by n URLs, gives you O(n²). A million URLs means a trillion comparisons. Use a set() and you get O(1) per check, total O(n), problem disappears.

You can't see this bug in your code. It's a property of which data structure you picked. CS2 + CS4 fix this.

2. The OOM-killed dedup

You're dedup'ing scraped product records by (brand, sku). The list of seen tuples is 80M entries. Python's set carries ~200 bytes per entry. You just allocated 16 GB. Your laptop says no.

Solutions exist (Bloom filters, content hashing, sharded dedup) and the Production sub-path's "deduplication at scale" lesson covers them. But that lesson assumes you understand what a set is, why it's O(1), and what trade-off a Bloom filter makes. That's CS2.

3. The interview whiteboard

Almost any data-engineering or scraping role with an interview round will ask one of:

• "Implement BFS over this directed graph."
• "Detect a cycle in a linked list."
• "Find the top-K most frequent items in a stream."

These are LeetCode mediums. They map directly to scraper internals (BFS = how you walk a site; cycle detection = how you avoid infinite redirect loops; top-K = trending products). You either know them or you don't. Half a sub-path here plus 50,100 LeetCode problems gets you through.

4. Reading other people's scrapers

When you open Scrapy's source and see PriorityQueue, OrderedDict, set, defaultdict, you should think "yes, that's the right shape for that job" not "what's a priority queue and why is it here." CS2 + CS3 make Scrapy's internals legible.

What this sub-path is NOT

• Not a CS degree. No automata theory, no formal language hierarchy, no compiler design.
• Not LeetCode grinding. You will need to practice problems (CS5), but the goal is fluency, not 500-problem volume.
• Not language-specific. Examples are in Python (because that's the curriculum language), but the ideas are language-neutral.
• Not advanced algorithms. No max-flow, no segment trees, no advanced DP. Easy and medium concepts only.

What you'll learn here

Five short lessons:

1. This lesson: why this sub-path exists.
2. Data structures for scrapers: the seven you'll actually use.
3. Algorithms and traversal: BFS/DFS as the crawler's brain, plus the searches and sorts you'll reach for.
4. Big-O in practice: time and space complexity, the way it actually bites scrapers.
5. How to study this: the one course + 50,100 LeetCode problems plan.

By the end you won't be a CS PhD. You'll be the scraper who reaches for set instead of list, draws the crawl graph on a napkin before writing code, and reads O(n log n) and immediately knows whether that's fast enough for the workload in front of you.

Where to practice

Nothing to practice yet. Lessons CS2 through CS5 each have concrete exercises (small Python snippets, LeetCode problem suggestions, and reading lists). Start CS2 when you're ready.