What Is Web Scraping Legal Risk? Robots.txt, ToS and Compliance in 2026
Robots.txt is not a lock, ToS is usually a civil risk, and proxies do not rewrite the law. A plain-English compliance map for scrapers in 2026.
People ask “is web scraping legal?” and expect a yes or a no.
They get a shrug instead. Then they buy a proxy pool and assume the problem is solved.
It isn’t. Legal risk for scrapers sits in layers: what the page shows, what the site asks you not to do, what privacy law says about people in the data, and what your tools actually change. Miss one layer and a “public page” project can still land you a ban, a lawsuit, or a compliance headache.
This guide maps those layers in plain English for 2026. It is not legal advice. Laws vary by country and by facts. For anything high-stakes, talk to counsel who knows your jurisdiction and use case.
- Robots.txt is a crawler courtesy file, not an access lock (RFC 9309).
- Breaking a site’s terms is usually a civil or contract risk, not instant crime.
- Personal data triggers privacy duties even when it was public on the page.
- Proxies and antidetect browsers hide your IP. They do not rewrite the law.
Why “Is Web Scraping Legal?” Is the Wrong First Question
Start with what you collect and how you reach it, not with a binary legal label. Web scraping is just automated collection of data from websites. The same technique can power price monitoring of public product pages or vacuum people’s profiles without a lawful basis.
Those two jobs do not share the same risk. Treating them as one question is how teams talk themselves into bad habits.
A better first pass looks like this:
| What you’re doing | Usual risk flavour | First check |
|---|---|---|
| Public product pages, no personal data | Lower; ToS and rate limits still matter | Terms + robots.txt + politeness |
| Public pages that include names, emails, profiles | Privacy / data protection duties | Lawful basis + minimization |
| Logged-in, paywalled, or fake-account access | Contract + unauthorized-access theories | Stop and get counsel |
| Ignoring a cease-and-desist, then rotating IPs | Looks intentional | Do not “proxy around” a ban |
Keep that table in mind. The rest of this post explains each row without pretending one US case or one robots.txt line settles everything.
What Robots.txt Actually Means (RFC 9309)
Robots.txt is how a site asks crawlers which paths to skip. It is not a password wall.
The IETF published that idea as an Internet Standard: RFC 9309, the Robots Exclusion Protocol (checked October 2026). The file lives at /robots.txt, uses allow and disallow rules for user-agent groups, and tells well-behaved crawlers where not to go.
The RFC is blunt about the legal mistake people make. In the introduction it says these rules are not a form of access authorization. In the security section it adds that the protocol is not a substitute for real content security, and that listing a path in robots.txt actually makes that path more discoverable.
So what does robots.txt do?
- It signals the owner’s preferences to crawlers that choose to honour them.
- It can include crawl-delay style courtesy via related practices, and RFC 9309 itself recommends short caching (generally under 24 hours) so rules stay fresh.
- It does not replace login, paywalls, or HTTP authentication for paths you truly want locked.
Our take: honour robots.txt as default policy unless you have a documented reason and legal review that says otherwise. Ignoring it is not automatically a crime under the RFC. It is still a loud signal that you knew the site did not want that traffic.
Public Pages Are Not the Same as Permission
If anyone can load a URL in a normal browser without logging in, that page is public in the everyday sense. That matters for some unauthorized-access theories. It does not equal a free licence to copy, store, resell, or profile whatever sits on the page.
Think of three different gates:
- Public HTML. Visible without credentials. Lower barrier, still covered by terms and (sometimes) copyright or database rights.
- Authenticated space. Behind login, session cookies, or paywalls. Crossing that line without permission is a different risk class.
- Technical blocks. CAPTCHAs, WAF rules, IP bans. Bypassing them after a clear ban can look like intentional circumvention, even if the underlying page once was public.
RFC 9309’s own security advice lines up with this: if a path must stay private, protect it with real access control, not a disallow line. For scrapers, the mirror is true. If a site put the good stuff behind a login, that is a warning light, not a puzzle for your headless browser.
For ecommerce collectors, our guide on how to scrape ecommerce sites safely focuses on polite, public-surface collection. That “safely” is operational. It is not a courtroom shield.
Terms of Service: Usually Civil Risk, Not Instant Crime
Most site terms ban automated collection, bulk download, or “scraping.” Agreeing to those terms (by account signup, clickwrap, or sometimes browsewrap arguments) can create a contract fight, not an automatic criminal case.
That distinction matters. Contract risk still hurts: injunctions, damages, account termination, vendor cutoffs. It is just a different bucket from “hacking” statutes.
US litigation around LinkedIn and hiQ Labs is a useful illustration, not a universal rulebook.
After years of fighting, LinkedIn announced in December 2022 that hiQ agreed to a permanent injunction against scraping LinkedIn and against using fake accounts there.
Commentary from counsel who followed the case (Fenwick, December 2022) notes that later stages leaned on contract and unfair-competition theories. Earlier Ninth Circuit rulings had narrowed how the federal Computer Fraud and Abuse Act (CFAA) applies to purely public data.
In plain English: even when a CFAA claim looks weak for public pages, a ToS claim can still be strong, especially if you created accounts, accepted terms, or kept scraping after a cease-and-desist.
Do not treat “it’s only ToS” as “it’s fine.” Treat it as “this may be civil, and civil still stops companies.”
Personal Data Raises the Stakes Fast
The moment your dataset can identify a person, privacy law joins the chat.
Under the EU GDPR’s Article 4 (checked October 2026 on EUR-Lex / UK legislation mirrors), personal data is any information relating to an identified or identifiable natural person. That includes names, ID numbers, location data, and online identifiers. “Processing” includes collection and storage, automated or not.
“But it was public” is a weak shield here. Public visibility can affect expectations and risk, yet GDPR-style regimes still ask for a lawful basis, a clear purpose, data minimization, security, and ways for people to exercise rights.
The UK Information Commissioner’s Office (ICO) explains legitimate interests as a three-part test: identify a legitimate interest, show the processing is necessary for it, and balance it against the person’s rights (ICO legitimate interests guidance, updated March 2026, checked October 2026). If a less intrusive method gets the same result, necessity fails.
The ICO has also written specifically about web scraping personal data to train generative AI.
In that context it treats legitimate interests as the only realistic lawful basis among the usual list, and only when the three-part test is actually passed. It calls large-scale scraping for model training high-risk and often “invisible” to the people involved.
That chapter is about gen-AI training, not about every scraper. Use it as an official warning that scraped personal data is regulated processing, not as a free pass for price bots.
Common mistake: copying profile fields “just in case” and only later asking whether you needed them. Minimization works the other way around. Collect less first.
What US Courts Have Said, at a High Level
US law on scraping is unsettled and fact-specific. This section is a map of themes from public court materials and reputable summaries, not a prediction for your case.
Two landmarks show up in almost every explainer:
- Van Buren v. United States (U.S. Supreme Court, 2021). The Court held that “exceeds authorized access” under the CFAA means reaching areas of a computer (files, folders, databases) that are off-limits, not merely using permitted access for a bad purpose. That cut back broad “policy violation equals CFAA crime” theories. Opinion: Supreme Court PDF (checked October 2026).
- hiQ Labs v. LinkedIn (Ninth Circuit, reaffirmed 2022 on remand). The Ninth Circuit’s public-facing rulings raised serious questions about using the CFAA to stop scraping of data that is generally publicly available. The dispute later ended with a stipulated permanent injunction against hiQ’s scraping and fake-account conduct on LinkedIn (LinkedIn announcement, December 2022). Contract claims remained central in the endgame.
Reputable case commentary (Fenwick / The Recorder, December 2022) summarizes practical rules of thumb that match how many US practitioners talk about the space today: collecting only public data without a contractual bar is unlikely to be a CFAA violation; creating fake accounts to reach logged-in data invites contract liability and potentially more; a wide middle gray area remains.
Warning: other circuits, statutes, state claims, copyright, and unfair competition theories can differ. Courts have disagreed on edges of this problem. If your business depends on aggressive collection, get counsel. Do not screenshot one blog post and call it clearance.
What Proxies and Antidetect Browsers Do Not Change
A proxy hides your origin IP. An antidetect browser reshapes fingerprints. Neither rewrites a contract, a privacy statute, or a court order.
Teams buy residential pools because sites block datacenter ranges. That is a real operational reason why web scraping needs proxies. It is not a compliance strategy.
The hiQ consent judgment language published on CourtListener even speaks in terms of access “directly or indirectly through a third party, intermediary, or proxy.” You do not need to memorize the filing. You need the intuition: swapping the IP after you were told to stop rarely looks like good faith.
Same story for antidetect tools. They can help separate account sessions and reduce fingerprint collisions. They do not bless fake identities created to reach gated data. For tool shopping that stays on the operational side, start from the proxy directory and pair it with a written collection policy, not the other way around.
Our take: use proxies to manage rate limits and geography on projects you already believe are allowed. Never use them as the plan for projects you already know are banned.
A Practical Compliance Checklist for Scrapers
Boring checklists beat clever excuses. Run this before you scale a new target.
- Write the purpose in one sentence. Price monitoring for SKUs is different from building a people database.
- Classify the fields. Flag anything that identifies a person. Drop fields you cannot justify.
- Prefer official APIs or licensed feeds when the publisher offers them.
- Read the terms and robots.txt. Save copies with the date checked. Decide, in writing, what you will honour.
- Stay on public surfaces. No fake accounts, no shared credentials, no “borrowed” sessions.
- Rate-limit and cache politely. RFC 9309’s crawler caching guidance is a courtesy cue: keep robots rules fresh and avoid hammering origins.
- Identify your bot when asked (clear User-Agent with a contact URL is the RFC 9309 spirit).
- If personal data is involved, pick and document a lawful basis. Under UK GDPR framing, that often means a recorded legitimate interests assessment.
- Secure the store. Access control, retention limits, and a delete path.
- Do not “proxy around” a cease-and-desist. Pause and escalate.
- Call counsel for social PII, health or finance data, children, or anything behind authentication.
Large jobs add operational stress on top of legal stress. Our notes on web scraping at large scale cover pacing and architecture. Pair that with this checklist so speed does not outrun judgment.
For the networking layer itself, knowing what proxies are helps you separate “IP hygiene” from “permission.” They solve different problems.
Frequently Asked Questions
The Bottom Line for Scrapers in 2026
Web scraping is a technique. Legal risk is a stack.
Robots.txt tells you what crawlers are asked to skip. Terms of service tell you what you may have agreed not to do. Privacy law cares when people appear in the payload.
Court fights in the US have narrowed some CFAA theories for public data while leaving contract claims and fake-account tactics very much alive.
Our take: the teams that stay out of trouble treat “can we request this URL?” and “should we keep this field?” as separate questions. Proxies belong to the first. Compliance belongs to both.
Next step: write a one-page collection policy for your current project, run the checklist above, and only then tune your pool. If the policy already feels shaky, fix the project before you fix the IP.
