Robots.txt Tester: Validate Allow and Block Rules

How to use the robots.txt tester

1. Paste your robots.txt content into the text area.
2. Enter the URL path you want to check (for example /admin/).
3. Type the user-agent string to test against (like Googlebot or *).
4. Read the result instantly: it shows whether the path is allowed or blocked, the matching rule, and any syntax warnings.

How the robots txt tester evaluates rules

This tool follows RFC 9309, the current standard for robots.txt. When you test robots txt rules, the parser groups directives by User-agent blocks. It selects the group whose agent name equals your user-agent token (case-insensitive, so Googlebot matches googlebot but a group named Specific does not match a crawler called SpecificBot). If several groups name the same agent, their rules are combined, and if no group names it the * group is used.

Within the chosen group, every Allow and Disallow path is matched against your URL path. The * character acts as a wildcard matching zero or more characters, and $ anchors the pattern to the end of the path. The rule with the longest matching pattern wins. If an Allow and a Disallow tie in length, Allow takes precedence. If no rule matches, the path is allowed.

Using this tool as a robots txt validator

Beyond checking single paths, this page works as a robots txt validator. As it parses your file, it collects syntax warnings for common mistakes:

• Lines that do not start with a known directive like User-agent, Allow, Disallow, Sitemap, or Crawl-delay.
• Directives missing a colon separator.
• Empty values where a path or agent name is expected.
• Rules that appear before any User-agent line, which parsers ignore.

Fixing these warnings ensures search engines interpret your file exactly as you intend.

Common robots.txt patterns

Here are typical patterns you can paste into the robots.txt tester to verify behavior:

Note that /*.pdf$ uses * to match any sequence including slashes, so it matches /dir/file.pdf. Always test edge cases with the tool rather than assuming.

RFC 9309 vs legacy parsers

Some older parsers, such as Python's urllib.robotparser, apply the first rule in file order that matches instead of the longest one and do not understand * and $ wildcards. RFC 9309 standardised the longest-match rule and the wildcards, and Google follows it, but smaller bots may still use legacy logic. This tester applies RFC 9309, so it can disagree with such parsers on files that mix Allow and Disallow rules.

If you need to generate a fresh file from scratch before testing, use a dedicated tool to build one, then paste the output here to validate every path.

Practical tips for testing

Always test both the exact path and variations. For instance, if you disallow /private, also test /private/ and /privatePage because /private as a prefix matches all three. Use the wildcard * intentionally when you need broader matching, and anchor with $ when you mean an exact file extension.

Remember that robots.txt is a voluntary standard. Malicious bots ignore it entirely, so never rely on it to protect sensitive data. Use proper authentication instead.

The tester only reads the text you paste: it does not fetch your live file. It matches user-agent names as exact tokens, so it does not apply Google-specific fallbacks such as using a Googlebot group for Googlebot-News, and it ignores Crawl-delay and Sitemap lines when deciding access.

Worked examples