Toolkeel

Robots.txt generator

Write a robots.txt file without syntax mistakes, decide which folders search engines should skip, and choose whether AI companies may crawl your site.

  • Free
  • Private
  • No sign-up
Default for all crawlers
Default
Block these AI crawlers (optional)
Test it in the robots.txt tester
Stays on this device. Network requests since you started: 0 How to check this yourself
  1. Your text and dataRead from your device
  2. Worked on in this tabBy code running in your browser
  3. Ready to copyNothing is stored

Never sent to a server. See how to check

How to use it

  1. Choose the default and list the folders to block.

  2. Add your sitemap URL and choose any AI crawlers to block.

  3. Download robots.txt and upload it to your site root.

About this tool

Upload the file as robots.txt to the root of your site (for example public_html/robots.txt), so it is available at yoursite.com/robots.txt. It controls crawling, not indexing: a blocked URL can still appear in Google if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag.

AI crawlers: blocking GPTBot, ClaudeBot, Google-Extended and others asks those companies not to use your content for training. Blocking search-specific agents such as OAI-SearchBot or Claude-SearchBot may also stop your site appearing in their AI search answers, so consider the trade-off. Compliance is voluntary; reputable companies say they respect these rules.

Good to know

  • robots.txt is a request, not access control. Protect private areas with passwords.
  • Crawler names change; check each company's documentation for current tokens.

Frequently asked questions

Where do I upload robots.txt?

To the top-level folder of your website, so it is reachable at yoursite.com/robots.txt. On cPanel that is usually public_html.

Should I block AI crawlers?

It depends on your goals. Blocking training crawlers protects your content from being used to train models; blocking AI search crawlers may also reduce visibility in AI answers.

Can robots.txt hide private pages?

No. The file is public and well-behaved bots only. Use passwords or server rules for private content.

You might also need

Last updated 2026-10-20 by the Toolkeel team. How we build and check our tools.