Noindex Page Finder
Find pages hidden from search results
What Is the Noindex Page Finder?
A stray noindex can quietly take a page, or a whole section, out of Google. The Noindex Page Finder crawls your site and checks every page for both ways to block indexing: the meta robots tag and the X-Robots-Tag header. It shows the exact directive on each page, so you can tell a deliberate noindex on a thank-you page from an accident on a product page.
Key Features
Noindex Isn't Always a Mistake
Thank-you pages, logins, carts, internal search results and filtered category pages are often noindexed on purpose, and should be. The risk is the page that shouldn't be: a template that shipped from staging with noindex still on, or a plugin setting applied to a whole section. Read the list against what each page is for.
Headers Are Easy to Miss
The X-Robots-Tag header blocks indexing just like the meta tag, but it doesn't appear in the page source, so it's easy to overlook. This crawler requests every page directly and reads the header on all of them, not a sample.
robots.txt Is a Different Thing
robots.txt stops crawling, not indexing. A page blocked there can still show up in results if other sites link to it, just without a snippet. The report lists those URLs separately so the two are never confused.
Common Use Cases
- Checking a launch or migration for leftover staging noindex tags
- Confirming which sections are kept out of search on purpose
- Finding pages that point their canonical somewhere unexpected
- Auditing robots directives set by plugins or CDN rules
Frequently Asked Questions
Is the Noindex Page Finder free?
Yes. It's free with a ProxyHorizon account, and every account gets 25 runs a day shared across all our data tools. You can read about the tool without signing in, but running it needs an account because each run fetches live data.
Is a canonical to another URL the same as noindex?
No, but the effect is similar. A canonical pointing elsewhere tells Google to index the other URL instead, so those pages are listed separately as indexable but pointing away.
What if the site blocks plain requests?
Then the crawler loads pages in a real browser instead, which can't read response headers. The report says so, and only the meta robots tag is checked.
Does the crawler follow robots.txt?
Yes. It skips anything robots.txt disallows for all crawlers and slows down when the file sets a crawl delay. It requests pages directly, which is how search engines first see them. If a site blocks plain requests outright, it loads up to 100 pages in a real browser instead and the report says so.
Do you keep the crawl?
For 24 hours. The crawl behind a report is stored for a day so the other sitewide tools can reuse it for the same site and page limit, instantly and without crawling again. It only holds public pages, so anyone checking the same site that day gets the same crawl. Choose "Crawl again now" whenever you want fresh data. We also log which tool you ran and when, to apply the daily limit.
Crawling whole sites for SEO at scale?
Proxies for SEO monitoring