robots.txt Generator
Generate a robots.txt file with user-agent rules, allow/disallow paths, and a sitemap directive.
About this tool
Compose a robots.txt file for search engine crawlers: user-agent groups, allow and disallow path rules, and an optional sitemap URL. Download or copy the finished file for your site root.
What you get
- User-agent line (specific bot or * for all)
- Allow paths — one rule per line
- Disallow paths — one rule per line
- Sitemap URL directive when provided
- Copy or download as robots.txt
- Live preview as you edit paths
How to use
Set the user-agent (or * for all crawlers), add allow and disallow paths, enter an optional sitemap URL, then copy or download.
Related searches
- robots.txt generator
- robots txt maker
FAQ
How to create a robots.txt file?
Set user-agent, add allow/disallow paths, optional sitemap URL, then copy or download robots.txt for your site root.
How to block a folder in robots.txt?
Add a Disallow line with the path prefix—for example /admin/—under the relevant User-agent group.
Should I include sitemap URL in robots.txt?
Yes—a Sitemap: directive helps crawlers discover your index file without guessing the path.
What is the difference between allow and disallow?
Disallow tells crawlers not to fetch matching paths; Allow can override a broader Disallow for specific subpaths in the same group.
How to allow Googlebot but block other crawlers in robots.txt?
Create a User-agent: Googlebot group with your allow rules, then a User-agent: * group with stricter Disallow lines for everyone else.
By using this tool you agree to our Terms of Use and Privacy Policy. Results are provided as-is without warranty.