Legal & Ethical Scraping, Your Compass

Is robots.txt legally binding?

• AYes, ignoring it is a federal offence
• BNo, it's a voluntary convention. Ignoring it isn't illegal in itself, but it's the simplest fact a court (or a site's PR team) uses to argue you were operating in bad faith. Respecting it is a defensible default
• COnly in the EU
• DOnly for commercial use