robots.txt Generator
Generate a robots.txt file with allow/disallow rules and sitemap URL.
About this tool
Compose a robots.txt file for crawlers: user-agent groups, allow/disallow paths, optional crawl-delay, and a sitemap directive. Runs locally — download or copy the file.
What you get
- User-agent blocks with allow and disallow paths
- Sitemap URL line
- Copy or download as robots.txt
How to use
Set the user-agent (or * for all), add path rules, optional sitemap URL, then copy or download.
FAQ
How to create a robots.txt file?
Set user-agent, add allow/disallow paths, optional sitemap URL, then copy or download robots.txt for your site root.
How to block a folder in robots.txt?
Add a Disallow line with the path prefix—for example /admin/—under the relevant User-agent group.
Should I include sitemap URL in robots.txt?
Yes—a Sitemap: directive helps crawlers discover your index file without guessing the path.
What is the difference between allow and disallow?
Disallow tells crawlers not to fetch matching paths; Allow can override a broader Disallow for specific subpaths in the same group.
By using this tool you agree to our Terms of Use and Privacy Policy. Results are provided as-is without warranty.