Robots.txt Checker

Fetch and validate a site's robots.txt and test whether a URL is allowed for a crawler.

Continue your work

Robots.txt Generator — Create a new robots.txt.
XML Sitemap Validator — Check the sitemap listed in robots.txt.

Found it useful? Send it to a friend or teammate.

X Facebook LinkedIn WhatsApp Telegram Email

Rate this tool

0.0

0 ratings

5 stars 0
4 stars 0
3 stars 0
2 stars 0
1 star 0

Clear instructions

Find steps, examples and limitations below.

Use online

Open the tool in a supported web browser.

Free to use

No sign-up required. Tool-specific limits may apply.

How to use the Robots.txt Checker

Fetch and validate a site's robots.txt and test whether a URL is allowed for a crawler.

1 Enter the website address and click Check robots.txt.
2 Review the validation results and the rules grouped by user agent.
3 In Test a URL, enter a path and choose a crawler to see whether it may crawl it and which rule decides.

Example and practical tips

Testing /search for Googlebot on google.com shows “Blocked” by Disallow: /search. Testing /search/about shows “Allowed”, because the longer Allow rule wins.

Frequently asked questions

How does Google choose between rules?

The most specific (longest) matching rule wins. If an Allow and a Disallow rule are the same length, Allow wins.

Does robots.txt hide a page from Google?

No. It stops crawling, but a blocked URL can still appear in results if other sites link to it. Use a noindex tag to keep a page out of results.

Test a URL