Robots.txt Tester
Test any URL against robots.txt
What Is the Robots.txt Tester?
The Robots.txt Tester fetches a site's live robots.txt and tells you whether a crawler may access a URL, using the same rules Google does. You see the exact rule that decided it, plus a table showing which search and AI crawlers, from Googlebot and Bingbot to GPTBot and ClaudeBot, the site lets in.
Key Features
Robots.txt Rules Are Trickier Than They Look
A crawler obeys only the group with its most specific user-agent, falling back to * if none matches. Within that group the longest matching rule wins, and Allow beats Disallow on a tie. Wildcards and the $ end marker add more edge cases. That's why eyeballing the file often gives the wrong answer.
Which AI Crawlers Can Read Your Site
Many publishers now block AI training crawlers like GPTBot, Google-Extended and CCBot while letting search crawlers through. Some also block the bots that power AI search, like OAI-SearchBot and PerplexityBot, which can keep them out of AI answers. The crawler table shows the full picture for any URL in one run.
Common Use Cases
- Checking whether a page is blocked from Google
- Seeing which AI crawlers a site allows
- Verifying robots.txt changes before they go live
- Checking whether a site allows scraping of a section
Frequently Asked Questions
Is the Robots.txt Tester free?
Yes. It's free with a ProxyHorizon account, and every account gets 25 runs a day shared across all our data tools. You can read about the tool without signing in, but running it needs an account because each run fetches live data.
What if a site has no robots.txt?
Then every crawler may access every page. Google treats a missing file (any 4xx response) as no restrictions.
What if robots.txt returns a server error?
Google treats a 5xx error as "crawl nothing" until the file is reachable again, so a broken robots.txt can stop indexing.
Does robots.txt stop scrapers?
Only polite ones. Robots.txt is a convention, not a lock. Responsible scrapers respect it, and many sites back it up with bot protection.
Auditing sites or tracking SEO every day?
Proxies for SEO monitoring