How to test a robots.txt file (and fix the mistakes that block Google)

Test a URL in under a minute

1. Open the robots.txt tester.
2. Paste the contents of your robots.txt file. You can see it by adding /robots.txt to your domain, for example https://example.com/robots.txt.
3. Enter the path you want to check, such as /blog/my-post/, and pick the crawler (Googlebot, Bingbot or any user agent).
4. The tester shows allowed or blocked and highlights the exact rule that decided it.

The file is checked in your browser. Nothing is sent to a server, so you can test a draft before you publish it.

How Google decides which rule wins

Crawlers first pick the group that matches their name most specifically (a User-agent: Googlebot group beats User-agent: * for Googlebot). Inside that group, the rule with the longest matching path wins, whatever order the lines are in.

The five mistakes we see most

• A leftover Disallow: /. Staging sites often block everything, and the file gets copied to the live site at launch. One line hides the whole site from crawling.
• Blocking CSS and JavaScript. Google renders pages like a browser. If /assets/ is blocked, it may see a broken page and judge it poorly on mobile.
• Using robots.txt to remove a page from Google. A disallowed URL can still be indexed from links pointing to it. To keep a page out of results, allow crawling and add a noindex robots meta tag; Google has to crawl the page to see that tag.
• The wrong host. Rules apply only to the exact host, protocol and port where the file lives. A file on www.example.com does not cover shop.example.com.
• A typo in the directive. Dissallow or a missing colon is silently ignored, so the rule simply does not exist.

What happens when robots.txt cannot be fetched

Google treats a missing file (a 404 or other 4xx error, except 429) as "no restrictions" and crawls everything. A server error (5xx) is different: Google pauses crawling for the first 12 hours while it retries, then uses the last good copy for up to 30 days. If your site briefly showed errors at launch, that is why crawl reports may mention robots.txt problems for a few days.

Checking in Google Search Console

Google's old standalone robots.txt Tester was retired in late 2023. Search Console now has a robots.txt report (Settings, then robots.txt) that shows the files Google found for your hosts, when each was last crawled, and any errors. After fixing a file you can request a recrawl there. To check a single page, use the URL Inspection tool, which says whether crawling is allowed.

A sensible starting file

User-agent: *
Disallow: /cart/
Disallow: /search?

Sitemap: https://example.com/sitemap.xml

Block only what has no value in search (carts, internal search results, admin areas), and list your sitemap. The robots.txt generator builds a file like this from a few choices, and the sitemap generator creates the sitemap it points to. Google reads up to 500 KiB of the file; anything after that is ignored.

Written by the Toolkeel team at Sole Digital Media

Fact-checked against the sources above. We test every tool we mention. Editorial policy.