Robots.txt Generator: Build Your Robots Txt File

How to use the robots.txt generator

1. Choose a preset from the dropdown or select Custom to build your own rules.
2. If using custom mode, enter one or more user-agents (use * for all crawlers).
3. Add the paths you want to Disallow or Allow, one per line.
4. Optionally set a Crawl-delay and add your Sitemap URLs.
5. Click Generate to produce the robots.txt content, then copy it to your site root.

How the robots txt generator works

A robots.txt file tells search engine crawlers which parts of your site they may access. This robots txt generator builds that file by combining User-agent, Disallow, Allow, Crawl-delay and Sitemap directives according to RFC 9309.

When you pick a preset, the tool outputs a pre-written block. For example, the "Allow all" preset produces User-agent: * followed by an empty Disallow: directive. An empty Disallow value means no paths are blocked. If you choose "Block all", the output changes to Disallow: /, which prevents every crawler from accessing any URL on the domain.

In custom mode, each line you type in the user-agents field becomes its own User-agent: directive. All subsequent Allow, Disallow and Crawl-delay lines apply to every listed agent until the next group.

Common presets explained

The built-in presets cover the most frequent use cases when you need to create a robots txt file quickly.

You can start with a preset and then switch to Custom to fine-tune individual paths.

Syntax rules under RFC 9309

RFC 9309 formalised the robots.txt protocol in 2022. A few key rules affect how you write your file:

• Directive names are case-insensitive (User-agent and user-agent are the same), but the paths after them are case-sensitive, so /Admin/ and /admin/ are different rules.
• Each group starts with one or more User-agent lines followed by rule lines. A blank line separates groups.
• The * wildcard matches zero or more characters in a path. The $ anchor matches the end of a URL.
• Crawl-delay is not part of RFC 9309. It is a non-standard extension that Bing honours but Googlebot ignores, and Yandex no longer uses it either.
• Sitemap directives are global and can appear anywhere in the file, though placing them at the end is conventional.

This robots-txt-generator checks that paths start with a slash or a wildcard and that sitemap URLs are absolute, and it adds an empty Disallow: line when a group has no rules so the group is explicit.

Where to place the robots.txt file

The file must live at the root of your domain: https://example.com/robots.txt. Crawlers do not look in subdirectories. If your site uses HTTPS, the robots.txt must also be served over HTTPS.

On Apache or Nginx you can simply upload the text file to the document root. On platforms like WordPress, a virtual robots.txt is generated dynamically, but you can override it by placing a physical file or using a plugin.

After uploading, test the file with a tool like the robots.txt tester or generate an accompanying sitemap with the sitemap XML generator.

Mistakes to avoid

Even a small typo can cause unintended crawling behaviour. Watch out for these pitfalls:

• Missing leading slash. Paths must begin with /. Writing Disallow: admin instead of Disallow: /admin/ will not match anything.
• Blocking CSS and JS. Modern search engines render pages. If you disallow /assets/ where stylesheets live, your site may appear broken in search results.
• Using robots.txt to hide pages. The file is public. Anyone can read it. Use authentication or noindex meta tags for sensitive content.
• Multiple conflicting groups. A crawler obeys only the group that names it most specifically and ignores the * group, so a bot-specific group must repeat any rules you also want it to follow. Under RFC 9309, separate groups that name the same bot are combined.
• Forgetting the sitemap line. Adding Sitemap: https://example.com/sitemap.xml helps crawlers discover your pages faster.

Worked examples