Fetch and validate a site's robots.txt and test whether a URL is allowed for a crawler.
How to use Robots.txt Checker
Enter the website address and click Check robots.txt.
Review the validation results and the rules grouped by user agent.
In Test a URL, enter a path and choose a crawler to see whether it may crawl it and which rule decides.
Worked example
Testing /search for Googlebot on google.com shows “Blocked” by Disallow: /search. Testing /search/about shows “Allowed”, because the longer Allow rule wins.
How does Google choose between rules?
The most specific (longest) matching rule wins. If an Allow and a Disallow rule are the same length, Allow wins.
Does robots.txt hide a page from Google?
No. It stops crawling, but a blocked URL can still appear in results if other sites link to it. Use a noindex tag to keep a page out of results.
Put this guide into practice
Robots.txt Checker — Fetch and validate a site's robots.txt and test whether a URL is allowed for a crawler.