GlossaryWeb ScrapingIntermediate

Crawl Rate

Crawl rate is how fast a crawler sends requests to a website, usually measured in requests per second. Keeping it reasonable avoids overloading servers and triggering blocks.

Last updated June 8, 2026

Definition

Crawl rate refers to the speed and frequency at which a bot or scraper makes requests to a target site, often expressed as requests per second or the delay between requests. It is one of the most important levers for scraping responsibly and avoiding detection.

How crawl rate works

A scraper can be throttled by adding a delay between requests, limiting concurrent connections, or honoring a Crawl-delay directive in robots.txt. Search engines like Google adjust their crawl rate dynamically based on how quickly a server responds.

Why it matters for scraping and anti-bot

  • An aggressive crawl rate trips rate limiting and anti-bot systems, leading to CAPTCHAs or IP bans.
  • Spreading requests across rotating proxies lets you maintain throughput while keeping the per-IP rate low.
  • A polite crawl rate reduces server load, lowering the chance of being flagged as malicious traffic.

Balancing speed against stealth is central to large-scale scraping: too fast and you get blocked, too slow and the job never finishes.

Examples

1

Sending one request every two seconds to avoid overloading a server

2

Googlebot slowing its crawl rate when a site responds slowly

3

Spreading 100 requests per second across 100 rotating proxy IPs

Common Use Cases

Throttling requests to avoid rate limits and bans
Tuning concurrency for large-scale scraping jobs
Honoring a crawl-delay set in robots.txt
Distributing load across rotating proxies to stay under per-IP limits

Frequently Asked Questions

It depends on the target, but a common starting point is one request every few seconds per IP, then adjusting based on server responses and any robots.txt crawl-delay.
Rotating proxies spread requests across many IPs, so the rate seen by the target per IP stays low even when your total throughput is high.
You are likely to hit rate limits, receive CAPTCHAs or 429 errors, and risk getting your IPs banned by anti-bot systems.

Keep Learning

All terms

Web Scraping

Web scraping is the automated extraction of data from websites — fetching pages programmatically and parsing their content into structured data.

Read definition

Rotating Proxy

A rotating proxy automatically assigns a different IP address from a pool for each request or on a set interval, spreading traffic across many IPs to avoid blocks.

Read definition

IP Rotation

IP rotation is the practice of automatically cycling through multiple IP addresses so that successive requests originate from different IPs.

Read definition

Rate Limiting

Rate limiting restricts how many requests a client can make in a given time, and it is one of the most common defenses scrapers must work around.

Read definition
Back to Glossary