Robots.txt Generator
Generate a correct robots.txt file with allow/disallow rules, crawl-delay and your sitemap, ready to drop on your server.
Rules
robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlSave this as robots.txt in your site root (e.g. https://example.com/robots.txt).
The robots.txt generator builds a correct robots.txt file for your site in seconds. Set your allow and disallow rules, add your sitemap, optionally block AI training crawlers, and copy a clean file that controls how search engines crawl your site.
How to use the Robots.txt Generator
- 1Choose whether to allow all crawlers by default.
- 2Add the paths you want to disallow, such as /admin/ or /cart/.
- 3Add your sitemap URL and any crawl delay.
- 4Copy the generated robots.txt and upload it to your site root.
What robots.txt controls and what it does not
Robots.txt tells crawlers which paths they may or may not crawl, which helps you save crawl budget and keep private or duplicate sections out of the crawl. It does not reliably keep a page out of the index on its own, for that you use a noindex tag. Used correctly, it is a simple but important part of technical SEO.
Common robots.txt mistakes to avoid
The most damaging mistake is accidentally disallowing your whole site with Disallow: / on a live site, which can remove it from search. Other pitfalls include blocking CSS or JavaScript that Google needs to render the page, and forgetting to add the sitemap line. This generator produces correct syntax so these errors are easy to avoid.
Should you block AI crawlers?
You can choose to block AI training crawlers like GPTBot and CCBot if you do not want your content used for model training. Note that blocking them is a separate decision from being visible in AI search, and many sites choose to stay open so they can be cited. The generator lets you add these blocks with one toggle.
Frequently asked questions
Is the robots.txt generator free?
Yes, it is free and needs no signup. Configure your rules and copy the file.
Where do I put robots.txt?
Upload it to the root of your domain so it is reachable at https://yoursite.com/robots.txt.
Does robots.txt remove a page from Google?
No. Robots.txt controls crawling, not indexing. To keep a page out of the index, use a noindex meta tag and allow the page to be crawled so Google can see it.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap line helps search engines discover your sitemap and crawl your pages more efficiently.
Can I block ChatGPT and other AI crawlers?
Yes. The generator can add rules for common AI crawlers such as GPTBot, ClaudeBot and CCBot if you choose to block them.
Want the full picture?
OneClickSEO tracks rankings, audits your site, monitors backlinks and your visibility across every AI search engine, all in one place.