robots.txt Tester
Paste your robots.txt and instantly test if any URL path would be allowed or blocked for a given crawler.
Enter a website URL and click Fetch to load its robots.txt, then test any URL path against it.
About this tool
robots.txt Tester lets you paste any robots.txt file and immediately test whether a given URL path would be crawled or blocked for a specific user-agent (Googlebot, Bingbot, all bots, etc.). The parser follows Google's robots.txt specification: longer (more specific) paths take precedence over shorter ones, and Allow rules beat Disallow rules when both match at the same path length. The matched rule is displayed so you can understand exactly why a URL is allowed or blocked. Use it to debug unexpected crawl exclusions, validate a new robots.txt before deploying, or check that private sections of your site are properly protected.
How to use
- Paste your robots.txt content into the text area (a sample is pre-loaded).
- Enter the URL path you want to test (e.g. /admin/dashboard).
- Select the user-agent to test as, then read the instant result and matched rule.
Related Tools
Trace the full redirect chain for any URL and see each HTTP status code.
Fetch any URL and verify the canonical tag — detect self-referential and cross-domain canonicals.
Inspect the HTTP response headers of any URL — cache policy, security headers, content encoding, and more.
FAQ
How does robots.txt path matching work?
Paths are matched from the start of the URL path. A rule of /admin/ blocks everything under /admin/. The asterisk (*) is a wildcard matching any sequence of characters. A dollar sign ($) at the end of a rule matches the end of the URL. When multiple rules match, the longest (most specific) path wins. If Allow and Disallow rules tie on length, Allow wins per Google's specification.
Does robots.txt prevent pages from appearing in search results?
No — blocking crawling in robots.txt prevents Google from reading the page, but does not remove it from the index. Google can still index a URL it has never crawled if other sites link to it. To prevent a page from appearing in search results, use the noindex meta robots tag or the X-Robots-Tag HTTP header instead.
What is the x-default user-agent in robots.txt?
The wildcard user-agent * applies to all crawlers not explicitly named. Most bots that respect robots.txt — including Googlebot and Bingbot — also respect rules set for *. However, named user-agents (e.g. User-agent: Googlebot) take precedence over * rules for that specific bot.
Is my robots.txt parsed in the browser?
Yes — all parsing and testing runs entirely in your browser using JavaScript. Your robots.txt content is never sent to any server. The tool works offline after the page has loaded.