Robots.txt Generator
Build a valid robots.txt file with allow, disallow, and sitemap rules.
Ignored by Googlebot. Respected by Bingbot and others.
User-agent: * Disallow: /
About this tool
Robots.txt Generator creates a correctly formatted robots.txt file without requiring you to memorise the syntax. Choose the user-agent (all crawlers, Googlebot, Bingbot, or a custom value), add as many Allow and Disallow path rules as you need, set an optional crawl delay, and enter your sitemap URL. The live preview updates with every change so you can spot mistakes before publishing. When finished, download the file directly or copy the content to paste into your site root.
How to use
- Select the user-agent or enter a custom crawler name.
- Add Allow and Disallow path rules one by one.
- Enter your sitemap URL and optional crawl delay, then copy or download the file.
Related Tools
Generate HTML meta tags including OpenGraph and Twitter Card markup.
Generate a valid XML sitemap with multiple URLs, priorities, and change frequency.
Fetch any URL and verify the canonical tag — detect self-referential and cross-domain canonicals.
FAQ
Where does robots.txt need to be placed?
The file must be placed at the root of your domain — for example https://example.com/robots.txt. Search engine crawlers check this exact URL before crawling your site, so placing it in a subdirectory will not work.
What does Disallow: / mean?
Disallow: / tells crawlers not to access any page on the site. This completely blocks indexing. Use it only during development or for staging sites you never want indexed.
Does robots.txt guarantee my pages won't be indexed?
No. Robots.txt is a directive, not a technical block. A page blocked in robots.txt can still appear in search results if another site links to it. To guarantee a page is not indexed, use the noindex meta tag or X-Robots-Tag header instead.
Can I have multiple User-agent sections?
Yes. Real-world robots.txt files often contain separate blocks for different crawlers — for example separate rules for Googlebot, Bingbot, and * (all others). This generator creates one block at a time; paste multiple blocks together for the final file.
What is Crawl-delay?
Crawl-delay asks crawlers to wait a specified number of seconds between requests to reduce server load. Googlebot ignores this directive — use Google Search Console's crawl rate setting instead. Bingbot and other crawlers do respect it.