HireWebDeveloper.net

Scraping & automation: what is legal, what gets built

The legal lines around scraping and automation: ToS, robots.txt, rate limits, personal data, what this practice builds (n8n, scheduled jobs) and what it declines.

Automation of your own work: almost always fine

Connecting your own systems, CRM to email, orders to accounting, forms to Slack, through n8n, Make, or scheduled scripts is uncontroversial and is most of what the automation practice builds. You own the data, the accounts, and the process. The engineering care points are reliability (retries, failure alerts, the monitoring discipline), credentials hygiene (scoped API keys, never passwords in scripts), and rate limits (your APIs have them; respecting them is what keeps the integration alive).

Scraping other people's sites: the honest lines

The legal landscape is genuinely murky, court decisions have narrowed and widened it over the years, and it varies by jurisdiction, so the working rules are conservative: public data, honestly obtained, used respectfully is generally defensible; everything else is declined. Concretely: respect robots.txt and terms of service (violating either converts a gray area into a lawsuit-shaped one), rate-limit so you do not effectively attack the target, never scrape personal data at scale (privacy law attaches immediately), never strip copyrighted content into your own property, and never scrape login-walled content. The hiQ-vs-LinkedIn saga made headlines; the safer summary is that winning a scraping fight is expensive even when you are right.

What gets built here, and what gets declined

Built: monitoring your own listings, syncing your own multi-channel inventory, aggregating feeds you have API access to, scheduled report pulls, internal workflow automation, and integrations via official APIs (the integrations practice, where an API exists, it beats scraping every time: stabler, legal, and the vendor maintains it). Declined: scraping behind logins, bypassing anti-bot measures, bulk personal-data harvesting, republishing others' content, with the reasons stated, not just a refusal. The automation audit finds where official APIs exist; the quote prices the safe version of the job.

Quarterly, and only when the numbers move

Get the rate report before you negotiate.

Updated rate bands across the major stacks and regions, plus what changed and why. No other email.

Read the current edition →

Ready to put this guide to work?

Six-question brief, scoped quote within two business days, and every term from the contract guide, in the actual contract.