Robots.txt Generator
Build a robots.txt file with crawler rules, blocked paths, AI bot controls and your sitemap.
Blocking these asks AI companies not to use your pages. Normal search results in Google and Bing are not affected.
Bing and Yandex respect this; Google ignores it.
Upload the file to the root of your site so it opens at . robots.txt is a request to crawlers, not access control: use passwords to protect private pages.
Continue your work
- Domain Age Checker — Check the registration details of the site you are configuring.
- Open Graph Meta Generator — Create social-sharing tags for the pages you allow crawlers to index.
Rate this tool
0.0
0 ratings
- 5 stars 0
- 4 stars 0
- 3 stars 0
- 2 stars 0
- 1 star 0
Share this tool
Found it useful? Send it to a friend or teammate.
Report an issue
Something broken or not quite right? Tell us and we will look into it.
Clear instructions
Find steps, examples and limitations below.
Use online
Open the tool in a supported web browser.
Free to use
No sign-up required. Tool-specific limits may apply.
How to use the Robots.txt Generator
A robots.txt file tells search engines and other crawlers which parts of your website they may visit. This generator builds a correct file for you: choose whether crawlers are allowed, list the folders to keep out of search, add exceptions, block AI training crawlers such as GPTBot, ClaudeBot and Google-Extended, set an optional crawl delay and point crawlers to your sitemap. The file updates as you type, and you can copy it or download it ready to upload.
- 1 Enter your website address. The sitemap line is filled in automatically; edit it if your sitemap lives elsewhere.
- 2 Choose the default rule, list any paths to block (one per line), add allowed exceptions and tick the AI crawlers you want to block.
- 3 Copy or download robots.txt and upload it to the root of your site, so it opens at yourdomain.com/robots.txt. Test it in Google Search Console.
Example and practical tips
An online shop might block /cart/, /checkout/ and /search so crawlers spend their time on product pages, allow everything else, block GPTBot and CCBot to opt out of AI training, and add https://shop.example.com/sitemap.xml. A staging site should use "Block everything" so it does not appear in search results.
Frequently asked questions
Does robots.txt hide pages from Google?
It asks crawlers not to visit those paths, but a blocked page can still appear in search results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex tag, or protect the page with a password.
Will blocking AI crawlers affect my search rankings?
No. Bots like GPTBot, ClaudeBot and Google-Extended are separate from the crawlers that build search results. Blocking them opts your content out of AI training by the companies that honor robots.txt, while Googlebot and Bingbot can still index your site.
Is robots.txt a security feature?
No. It is a public file that polite crawlers follow voluntarily. Anyone can read it, so never list secret URLs in it, and protect private areas with authentication instead.
Where do I put the file?
In the root folder of your domain, so it is available at https://yourdomain.com/robots.txt. Each subdomain needs its own robots.txt file.