Robots.txt Generator
Generate a valid robots.txt: crawl rules per bot, sitemap declaration, presets for WordPress and common CMSs, and AI crawler controls. Free.
robots.txt
The result appears here as you type.
What this tool is for
Robots.txt tells crawlers where not to go. It is four lines of syntax with two persistent traps: it controls crawling and not indexing, and a blocked page can still appear in results if other sites link to it. Presets cover the usual platform paths, and AI crawler rules are included so you can decide that question deliberately.
How to use it
- Pick a preset or start from an empty file.
- Add disallow rules per user agent.
- Declare your sitemap URL: it belongs here and is frequently forgotten.
- Copy the result to the root of your domain as /robots.txt.
Frequently asked questions
Does robots.txt keep a page out of Google?
No. It stops crawling, not indexing. A blocked URL that other sites link to can still be listed, without a description, because the crawler was never allowed to read it. To keep a page out of results, allow crawling and use a noindex meta tag.
Should I block AI crawlers?
It is a business decision, not a technical one. Blocking GPTBot, ClaudeBot and others keeps your content out of training data and also out of the answers AI assistants give about your topic, which is a growing source of referrals.
Where does the file go?
The root of the host: https://example.com/robots.txt. A file in a subdirectory has no effect, and each subdomain needs its own.