Robots.txt Generator
Build a valid robots.txt file with allow, disallow, and sitemap rules.
Ignored by Googlebot. Respected by Bingbot and others.
User-agent: * Disallow: /
What is Robots.txt Generator?
Robots.txt Generator creates a correctly formatted robots.txt file without requiring you to memorise the syntax. Choose the user-agent (all crawlers, Googlebot, Bingbot, or a custom value), add as many Allow and Disallow path rules as you need, set an optional crawl delay, and enter your sitemap URL.
The live preview updates with every change so you can spot mistakes before publishing. When finished, download the file directly or copy the content to paste into your site root. All generation runs in your browser — no data sent to any server.
Common Use Cases
- Blocking staging, admin, or private pages from being indexed
- Pointing crawlers to your sitemap.xml location
- Setting a crawl delay for bandwidth-sensitive servers
- Restricting specific bots from accessing your site
- Preparing a complete robots.txt before a new website launch
How to Use Robots.txt Generator
- Select the user-agent or enter a custom crawler name.
- Add Allow and Disallow path rules one by one.
- Enter your sitemap URL and optional crawl delay, then copy or download the file.
Related Tools
Generate HTML meta tags including OpenGraph and Twitter Card markup.
Generate a valid XML sitemap with multiple URLs, priorities, and change frequency.
Fetch any URL and verify the canonical tag — detect self-referential and cross-domain canonicals.
FAQ
Where does robots.txt need to be placed?
The file must be placed at the root of your domain — for example https://example.com/robots.txt. Search engine crawlers check this exact URL before crawling your site, so placing it in a subdirectory will not work.
What does Disallow: / mean?
Disallow: / tells crawlers not to access any page on the site. This completely blocks indexing. Use it only during development or for staging sites you never want indexed.
Does robots.txt guarantee my pages won't be indexed?
No. Robots.txt is a directive, not a technical block. A page blocked in robots.txt can still appear in search results if another site links to it. To guarantee a page is not indexed, use the noindex meta tag or X-Robots-Tag header instead.
Can I have multiple User-agent sections?
Yes. Real-world robots.txt files often contain separate blocks for different crawlers — for example separate rules for Googlebot, Bingbot, and * (all others). This generator creates one block at a time; paste multiple blocks together for the final file.
What is Crawl-delay?
Crawl-delay asks crawlers to wait a specified number of seconds between requests to reduce server load. Googlebot ignores this directive — use Google Search Console's crawl rate setting instead. Bingbot and other crawlers do respect it.
How do I block Googlebot from crawling a specific folder?
Select Googlebot as the user-agent, then add a Disallow rule with the folder path, e.g. Disallow: /admin/. This tells Googlebot not to crawl any URL starting with /admin/. Use the live preview to verify the syntax before downloading.