Ethical Framework for Scraping Decisions

Not legal advice. Engineer's reflection.

The previous three lessons covered what scraping law permits and forbids. Ethics is the layer above: of the legal projects, which ones SHOULD you take?

This is the final lesson of the curriculum. The technical knowledge you've built is enough to scrape almost anything. The remaining question is which of those things you want your work to be associated with.

Why ethics matters separately from law

Three reasons not to collapse ethics into law:

1. Law lags reality. Privacy law was non-existent in 2010; now it's foundational. AI-training scrapes are legally ambiguous now; in five years they won't be. Operating only at the legal minimum means operating with regular surprises.

2. Your reputation tracks ethics, not law. Your peers, future employers, and your future self assess what you did, not what you could have argued was permitted.

3. Legal protection assumes good lawyers. The honest scraper with a defensible posture wins more arguments than the technically-clever scraper with a creative defense.

The four-question framework

For any scraping project, ask:

1. Is the data subject a willing participant?

A business publishing product prices on its website is implicitly participating in commerce. A scientist publishing a paper is sharing knowledge. An individual on a forum probably didn't expect their words extracted into a corpus.

The further from "willing participant," the more careful you should be. Personal data (lesson 82) is the strongest case; even non-personal data has gradations.

2. Does the data subject suffer harm from your use?

Harm can be:

• Privacy harm, exposure of personal information.
• Reputational harm, content surfaced in ways the subject didn't intend.
• Economic harm, competitor undermining the source.
• Cost harm, heavy scraping increasing the source's hosting bills.
• Trust harm, users avoiding the source because they fear being scraped.

If the answer is "they're materially worse off because I scraped," your posture is weaker even if the law permits it.

3. What value do I create?

Scraping is value-positive when:

• Consumers get better information (price comparison, product reviews).
• Public-interest research gets cheaper.
• A dysfunctional market becomes more efficient.
• Knowledge that was locked behind unnecessary friction becomes accessible.

Scraping is value-neutral or negative when:

• The output is just spam fodder.
• It enables harassment or manipulation.
• It only redistributes existing content without adding to it.
• It's a vehicle for arbitrage that doesn't improve any outcome.

Not every project needs to save the world. But "what's the value?" is a question worth asking before committing months to a project.

4. Would I be embarrassed if the target's lawyer / users saw exactly what I did?

This is the gut check. The fully transparent version:

• "Here's my User-Agent identifying the project."
• "Here's the rate I scraped at."
• "Here's the data I collected and what I did with it."
• "Here are the URL paths I respected and the ones I didn't."

If the answer is "I'd be fine with all that being public," your project is on solid ground. If you'd wince at any part, that's the part to address before shipping.

The "would I be comfortable explaining this to a non-technical friend?" test

The bluntest version of question 4: can you describe what you're doing in plain English to a friend who isn't a developer, and have them respond "yeah, that seems fine"?

• "I scrape prices on public e-commerce sites to make a price comparison tool" → fine.
• "I scrape job postings and email candidates" → starts to feel intrusive.
• "I scrape personal social media profiles and use them to train a model" → most non-techies recoil.

If the description feels like it needs a lot of caveats and "but technically...," the project might not pass the gut check.

Practical ethics, operational habits

Beyond project-selection ethics, daily habits matter:

Rate limiting as ethics

Even when legal, hammering a small site can run up its hosting bill. Cap your rate by what's polite for the target's scale. A big retailer can absorb 10 req/s; a small indie blog cannot.

Respect robots.txt unless you have a clear reason not to

robots.txt isn't law, but ignoring it is a strong signal of bad faith. Courts and journalists both notice. If you must ignore it, document why and limit the scope.

Identify yourself

Set a clear User-Agent: MyTool/1.0 (mailto:contact@example.com). Lets the target reach you. Lets you say in any later conversation, "I was always identifiable."

Don't bypass technical access controls

Paywalls, CAPTCHAs, IP blocks, these are the site saying "no." Bypassing them combines legal risk (CFAA-style claims) with ethical violations (overruling the site's express choice).

Honor delete/access requests

Even if you scraped public data, if a subject reaches out with a reasonable request, delete this, remove this listing, honor it. Stubbornness here loses arguments and rarely gains anything.

Minimize data

Collect what you need. Discard what you don't. Smaller dataset = smaller blast radius if anything goes wrong.

Projects to decline

Even when they pay well, decline projects where:

• Personal data is core: PII scraping at scale, especially of EU residents, is professional suicide-by-GDPR.
• The client wants you to bypass access controls: paywalls, CAPTCHAs explicitly designed to stop scrapers, login walls without account permission.
• The use case is harm-enabling: stalking, doxxing, harassment, election manipulation.
• You can't describe it without weasel words: "We just want all the data from this competitor's private API."
• The client is hostile to your ethics: "Just do it, don't overthink it."

Saying no to one project preserves your career for many future projects. Saying yes can end it abruptly.

The peer-respect test

Imagine submitting a description of your current project as a talk to your favorite scraping community. Would you be welcomed, or quietly judged?

Peer respect is the slowest-moving but most durable measure of whether your work is ethical. Engineers who do well over years have it; engineers who don't, often don't.

Documenting your ethics posture

For non-trivial projects, write down:

PROJECT: [name]
TARGET: [sites]
DATA NATURE: [facts? personal? mixed?]
DATA SUBJECTS: [businesses? individuals? aggregate users?]
HARM ASSESSMENT: [potential harms; mitigations]
VALUE CREATED: [for whom; how]
TRANSPARENCY: [User-Agent; identification; openness]
RATE LIMIT: [N req/sec; rationale]
ACCESS CONTROLS RESPECTED: [robots.txt; login walls; CAPTCHA]
DELETE/ACCESS POLICY: [how subjects can request changes]
GUT CHECK: [pass / fail / qualified]

Saved alongside the project. Six months later when someone asks "wait, did you scrape X?", the answer is in a document, not in your fading memory.

Personal practice over career

Some closing thoughts:

• Your career is long. The 30-year-old engineer's ethics affect the 60-year-old's options. Compound carefully.

• Bad projects feel small while you're doing them. They look damning in retrospect.

• Good projects feel small too. Many of your most ethical decisions will be silent: declining work, deleting data, slowing down a scrape. Few of these get noticed externally; all of them shape who you are.

• You can change directions. An engineer who scraped questionably in 2020 and rebuilt their practice ethically in 2024 is the engineer most other engineers respect, far more than the one who never scraped at all.

• Document your reasoning. The best engineers I've worked with leave a paper trail of why-they-did-what. Not for legal cover; for self-clarity.

Final exercise

For the most ambitious scraping project you'd consider taking on:

1. Run it through the four questions.
2. Write the one-page ethics doc.
3. Imagine reading it five years from now in court, in a peer review, and to a non-technical friend. All three responses should be acceptable to you.

If all three pass, proceed. If any fails, adjust the project or pick a different one.

Closing

You've now finished Sub-Path 4. The technical content covered Scrapy, Symfony, async, proxies, fingerprinting, CAPTCHAs, distributed crawling, monitoring, deployment, career, and legal. Each of those is a lifetime of depth available.

You don't need to master all of it. You need to know which parts apply to your work and to grow into them as your career unfolds.

The capstone (Sub-Path 5) is where you put it together: pick one project, build it end-to-end, ship it. That's where everything you've learned becomes something you can show.

Go build the thing. The internet needs more thoughtful scrapers and fewer hostile ones. You're now equipped to be the former.