How to Write a robots.txt That Works

The robots.txt file is the first thing a search engine reads on your site. It is a tiny plain-text file with outsized power — get it wrong and you can accidentally hide your whole site from Google. Here is how it works and how to write one that helps rather than harms.

What robots.txt is for

robots.txt lives at the root of your domain (example.com/robots.txt) and tells crawlers which parts of your site they may request. It is a set of polite instructions that well-behaved bots — Google, Bing and the rest — obey.

Two things it is not:

• It is not security. It only asks nicely. Malicious bots ignore it, so never use it to hide sensitive pages; protect those with authentication.
• It is not a way to remove pages from search. This is the most dangerous misunderstanding — see below.

The basic syntax

The file is a list of rules grouped by crawler. The essentials:

• User-agent: names the crawler the following rules apply to. * means all crawlers.
• Disallow: lists a path the crawler should not request. Disallow: /admin blocks the admin area; Disallow: / blocks the entire site.
• Allow: carves out an exception inside a disallowed area.
• Sitemap: points crawlers at your XML sitemap (use the full URL).

A sensible starter file looks like this: allow everything, keep bots out of your admin and private areas, and point them at the sitemap.

User-agent: *
Disallow: /admin
Allow: /

Sitemap: https://example.com/sitemap.xml

The mistake that hides your whole site

The single most common catastrophe is shipping this:

User-agent: *
Disallow: /

That one slash tells every crawler to request nothing. It is the default many frameworks use in development, and forgetting to change it on launch quietly removes the site from search. If "Google isn't showing my site", this is the first thing to check.

Disallow does not mean "remove from search"

Here is the subtle trap. Disallow stops a crawler from reading a page — but a blocked page can still appear in results (as a bare URL with no description) if other sites link to it. Worse, because Google cannot read the page, it cannot see a noindex tag on it either.

So the two tools do opposite-feeling jobs:

• To keep a page out of the index, let crawlers read it and add a noindex meta tag (or header). Do not block it in robots.txt.
• To save crawl budget on pages you do not care about (faceted filters, internal search results), Disallow is the right tool.

Blocking a page you are also trying to noindex is self-defeating — the crawler never sees the noindex.

What to block and what to leave open

• Block: admin panels, login pages, internal search result pages, duplicate faceted URLs, cart and checkout.
• Leave open: all your real content, your CSS and JavaScript (Google needs these to render and judge the page — blocking them hurts you), and your images if you want them in image search.

Test before you trust

A broken robots.txt fails silently; you only notice weeks later when traffic drops. So validate it: confirm it is reachable at the root, that it is not accidentally disallowing everything, and that a given URL is allowed for Googlebot as you intended.

Generating a correct robots.txt from a few choices, and checking an existing file to see exactly which URLs it blocks or allows, both run in your browser. Draft the file, verify your important pages are crawlable and your private ones are not, and confirm the sitemap line is present — before you deploy it to the one place every search engine looks first.