What is web scraping?
- Try the cheapest approach first: raw HTML, then the site’s own JSON endpoint, then a real browser.
- Blocks are rarely about one thing — IP reputation, headers, TLS fingerprint, JavaScript and behaviour all get scored.
- Proxies are necessary but not sufficient. Behaviour is the last mile.
- Residential proxies bill by bandwidth, so what you download matters more than how many pages you visit.
- Collecting public data is generally lawful, but terms of service, rate limits and data-protection law still apply.
The mechanics are simple — fetch a page, read some values. Everything difficult about scraping comes from doing it at scale, against sites that would rather you did not, without spending more than the data is worth.
Pick your approach
Cost and reliability differ by orders of magnitude. Start at the top and only move down when you must.
curl https://shop.com/p/123
# → <html>…<span class="price">£24.99</span>…The price is right there in the markup.
Server-rendered sites — news, blogs, older e-commerce. Always try this first.
CSS selectors are brittle. A layout change silently returns nothing.
Why your scraper gets blocked
100 requests to a well-defended retailer. Add one layer at a time and watch the success rate move.
Believable headers, randomised delays and session reuse. This is what a working pipeline looks like.
Illustrative rates for a strict target — your numbers depend entirely on the site. The shape is what matters: proxies alone do not solve it, and behaviour is the last mile.
The anti-bot ladder
Defences stack. Knowing which rung you are stuck on tells you exactly what to change.
- Header order & values
- TLS/JA3 fingerprint
- HTTP/2 frames
Send browser-accurate headers, or use a client that mimics a real TLS stack.
Cloudflare, DataDome and PerimeterX all run several of these at once and score you continuously. See bypassing Cloudflare for the practical detail.
Where proxies fit
Datacenter
Open pages, public APIs, and any target that does not check IP reputation. Fast and cheap — start here.
Best datacenter proxiesResidential
Retail, travel, ticketing and anything geo-sensitive. Billed per GB, so block images to keep the bill sane.
Best residential for scrapingMobile
Social platforms and the genuinely hostile targets. The most expensive option — escalate to it, do not start with it.
Mobile proxiesHow long will it take, and what will it cost?
Concurrency and delay decide both your runtime and how quickly a site notices you.
A defensible pace. Spread across rotating IPs this rarely trips simple rate limits.
Keeping it running
- 1
Monitor rows, not just status codes
A soft block or a layout change usually returns a perfectly valid 200 with nothing useful inside. Alert on sudden drops in extracted records.
- 2
Retry with backoff, and cap it
Transient failures are normal. Exponential backoff with a ceiling stops a struggling target turning into a runaway bandwidth bill.
- 3
Validate the shape of what you extract
Assert that key fields exist and look sane. A price field that starts returning empty strings should fail loudly, not fill your database with nulls.
- 4
Keep raw samples
Store a handful of raw responses per run. When a parser breaks at 3am, having the actual HTML is the difference between a five-minute fix and a day of guessing.
- 5
Cache aggressively
Do not refetch what has not changed. Conditional requests and a local cache cut both cost and detection risk.
Legal and ethical ground rules
Collect only public data
If it requires a login you are not authorised to use, or sits behind a paywall, do not take it.
Do not degrade the service
Rate-limit yourself. Knocking a small site over is both rude and the fastest way to legal trouble.
Respect terms and robots.txt
Not always binding, but ignoring them removes any good-faith defence you might have had.
Mind personal data
Names, emails and profiles fall under GDPR and similar laws. Have a lawful basis, or do not collect them.
Five expensive mistakes
- 1
Reaching for a browser first
Headless is the expensive last resort, not the default. Check the raw HTML, then the network tab, before you spin up Chromium.
- 2
Downloading everything
Images, fonts and video are billed identically to the data you want. Blocking assets routinely cuts residential proxy bills by 60–80%.
- 3
Fixed delays
A request precisely every 2.000 seconds is more obviously automated than no delay at all. Randomise around a mean.
- 4
Ignoring robots.txt and terms
Not always legally binding, but ignoring them removes your good-faith defence and is the fastest route to a legal letter.
- 5
Scaling before stabilising
Getting 100 pages reliably teaches you what breaks. Launching at 100,000 just breaks it faster and more expensively.
Test yourself
1A page shows prices but curl returns no price in the HTML. What is the most likely cause?
2Rotating residential proxies alone will get you past most anti-bot systems.
3Your scraper returns HTTP 200 but zero rows. What should you suspect first?
4Which is cheapest per page when the data is available all three ways?
5Adding a fixed 2-second delay between requests makes you look human.
Web scraping FAQ
1What is web scraping?
It is the automated collection of data from websites. A program requests pages the same way a browser does, then extracts the specific values you want — prices, listings, reviews — and stores them in a structured form such as a database or CSV.
2Is web scraping legal?
Collecting publicly available data is generally lawful in most jurisdictions, and courts have broadly supported that. What matters is how you do it: respect each site’s terms of service, do not access login-gated or private data you are not authorised to see, avoid overloading servers, and comply with data-protection law such as GDPR for anything personal.
3Why does my scraper get blocked?
Usually a stack of reasons rather than one. The IP has poor reputation, the headers do not match a real browser, requests arrive too fast or too regularly, or the site requires JavaScript you are not running. Fixing only the IP rarely fixes the problem.
4Do I need a headless browser?
Only when the data is not in the HTML. Try a plain request first, then look for the JSON endpoint the page itself calls. A browser costs 50–100 times more in resources, so reserve it for sites that genuinely need JavaScript or a signed request you cannot replay.
5Which proxies are best for scraping?
Match them to the target. Datacenter proxies for open pages and APIs where speed and cost matter. Residential for sites that check IP reputation. Mobile for the hardest social platforms. Most teams run a hybrid and escalate only where they must.
6How fast can I scrape without being noticed?
There is no universal number — it depends on the site and how many IPs you spread across. As a rule of thumb, sustained rates above roughly 20 requests per second from one identity look nothing like a human. Add jitter to delays rather than using a fixed interval.
7What is the difference between scraping and using an API?
A public API is offered deliberately, documented and stable, with clear terms. Scraping extracts data from the presentation layer, or from an undocumented internal endpoint. If an official API exists and covers what you need, use it — it is cheaper and far less fragile.
8How do I keep a scraper from breaking silently?
Monitor success rate and output volume, not just errors. A layout change or a block often returns a valid HTTP 200 with empty results. Alert on sudden drops in extracted rows, validate that key fields are present, and log a sample of raw responses for debugging.