What goes in a robots.txt file
robots.txt is a plain-text file at the root of a host (https://www.example.com/robots.txt) that tells crawlers which URLs they may fetch. It has three things you will normally use:
- Groups, each starting with
User-agent:lines naming the crawlers it applies to (*means every crawler without a group of its own). - Rules in each group:
Disallow: /pathblocks every URL whose path starts with/path;Allow:re-opens part of a blocked area. The longest matching rule wins. - Sitemap lines:
Sitemap: https://..., anywhere in the file, outside groups. They help every search engine find your sitemaps.
The generator writes the AI-crawler and other blocked groups first, then the * group with your rules, then the sitemaps. Order between groups does not matter to crawlers, but this layout is easy to read.
Worked examples
A typical shop
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sort=
Disallow: /*&sort=
Sitemap: https://shop.example.com/sitemap.xmlCarts and checkouts are private and endless; sort parameters create thousands of duplicate listing URLs. Product and category pages stay open. Note that /*?sort= only matches when sort is the first parameter, so the second rule covers it in any other position.
A staging site
User-agent: *
Disallow: /This stops well-behaved crawlers, but it is not access control: anyone can still open the pages, and URLs linked from elsewhere can still be indexed without content. Protect staging sites with a password (HTTP authentication) instead, which also keeps them out of search.
Opting out of AI training but staying in search
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow:GPTBot (OpenAI) and Google-Extended (Google's control for use in Gemini models) are turned away; Googlebot, Bingbot and every other crawler keep full access. Google-Extended is not a separate crawler: it is a token Google checks when deciding how content fetched by its normal crawlers may be used, so blocking it does not affect Google Search.
AI crawler tokens
| Token | Operator | What blocking it does |
|---|---|---|
GPTBot | OpenAI | Stops collection for training OpenAI models. OpenAI's search crawler (OAI-SearchBot) and user-triggered fetches (ChatGPT-User) are separate tokens. |
ClaudeBot | Anthropic | Stops collection for training Anthropic models. |
Google-Extended | Stops use in Gemini training and grounding. No effect on Google Search ranking or inclusion. | |
Applebot-Extended | Apple | Stops use in Apple's AI training; Applebot still crawls for Siri and Spotlight. |
CCBot | Common Crawl | Stops the open web archive that many AI datasets are built from. |
PerplexityBot | Perplexity | Stops indexing for Perplexity's answer engine. |
Meta-ExternalAgent | Meta | Stops collection for Meta's AI training. |
Bytespider | ByteDance | Stops ByteDance's crawler. |
robots.txt is a request, not an enforcement mechanism. The operators above say they honour it; others may not. Crawler names change, so check each operator's documentation from time to time.
Pitfalls
- Do not block pages you want removed from Google. Use
noindexinstead; a blocked page cannot show Google its noindex tag. - Do not block CSS and JavaScript that pages need to render.
- Paths are case-sensitive and must start with
/(or*). - A named group replaces the
*group for that crawler. If you addUser-agent: Googlebot, repeat in it every rule from*that should also apply to Googlebot. - Keep the file under 500 KiB; Google ignores the rest. Save it as UTF-8 plain text.
- One file per host and protocol. Subdomains need their own.