robots.txt Tester

What this tester does

It applies a robots.txt file to the URLs you give it, exactly as a well-behaved crawler would, and tells you for each URL whether that crawler may fetch it. It also names the group of rules that applied and the single line that decided the result, and it lists problems in the file itself: rules outside any group, unsupported directives such as Noindex or Crawl-delay, paths that do not start with a slash, and files above Google's 500 KiB limit.

Google removed its own robots.txt Tester from Search Console in December 2023 and replaced it with a report that shows which robots.txt files Google fetched. That report is useful, but it does not let you try a draft file or a list of URLs before you publish. This page does.

How to use it

1. Paste the contents of your robots.txt (it lives at https://your-site/robots.txt), or open a saved copy. A draft works just as well as the live file.
2. Choose the crawler. Search engines identify themselves with a short product token such as Googlebot or Bingbot; robots.txt groups are matched against that token, not against the full user-agent string.
3. Enter one URL or path per line. Full URLs and bare paths both work; only the path and query string matter, because a robots.txt file only governs its own host.

Results update as you type.

How crawlers read robots.txt

1. Pick one group

A file is made of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules. A crawler obeys only one group: the one naming it most specifically. If no group names it, it uses the User-agent: * group. If several groups name the same crawler, their rules are combined. Rules from the * group are not added to a specific group; this is the most common surprise. In the example below, Googlebot may crawl /private/, because its own group says nothing about it.

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /tmp/

Google's specialised crawlers fall back to their parent: if there is no Googlebot-Image group, Googlebot-Image follows the Googlebot group. A few Google crawlers, such as AdsBot-Google, ignore the * group and obey only rules that name them; this tester applies the general rule, so name them explicitly if you need to control them.

2. The longest matching rule wins

Within the group, every rule whose path matches the start of the URL path is a candidate. The candidate with the longest path wins, whether it is an Allow or a Disallow. If an Allow and a Disallow of the same length both match, the Allow wins. That is how you open one page inside a blocked folder:

User-agent: *
Disallow: /private/
Allow: /private/press-kit.html

3. Wildcards

* matches any run of characters, and $ at the end anchors the rule to the end of the URL. Everything else is a plain prefix match, and paths are case-sensitive.

Common mistakes

• Using robots.txt to remove pages from Google. Disallow stops crawling, not indexing. A blocked URL can still appear in results (without a description) if other sites link to it. To keep a page out of the index, let it be crawled and add <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header. Google stopped honouring Noindex: lines in robots.txt in 2019.
• Blocking CSS, JavaScript or images the page needs. Google renders pages like a browser. If it cannot fetch your scripts and styles, it may misjudge the page, for example as not mobile-friendly.
• Expecting the * group to add to a named group. It does not; see above. Repeat shared rules in every named group.
• Forgetting that each host and protocol has its own file. https://shop.example.com/robots.txt does not govern https://www.example.com/, and http and https are separate too.
• Serving errors for robots.txt. If the file returns a 4xx status, Google treats the site as having no restrictions. If it returns a 5xx status or times out, Google stops crawling the site for a while and then falls back to its last cached copy. Google caches robots.txt for up to 24 hours, so changes are not instant.
• Crawl-delay for Google. Googlebot ignores it. Bing and Yandex read it.

Sources

• RFC 9309: Robots Exclusion Protocol (IETF, 2022)
• How Google interprets the robots.txt specification
• Google's crawlers and their user-agent tokens