CanDoYa
All tools
Saved toolsYour favorites

Add to favorites

Saved in this browser only.
Private browser utility / SEO & Web

Free Robots.txt Generator

Runs entirely in your browser - no upload, no sign-up.

Live workspaceLocal processing
Quick starts

User-agent groups

Create one group for all crawlers or add separate rules for a named bot.

Crawler group 1
Group 1
Enter one path per line. Each path must begin with /.
Live preview

Generated robots.txt

1groups
3rules
1sitemaps
The generated directives are valid.
Share this tool
robots txt generator / browser utility
01 / Overview

What is a robots.txt generator?

A robots.txt generator builds the plain-text file that tells search engines and other well-behaved crawlers which URL paths they may request. Add user-agent groups, Allow and Disallow paths, and sitemap URLs, then copy or download standards-aware output without memorizing the directive syntax. Everything is generated locally in your browser.

02

How to use

  1. 01
    Choose a starting point

    Load the starter, open-site, WordPress, or block-all preset, or clear the form for a blank configuration.

  2. 02
    Set crawler rules

    Use * for all crawlers or enter a named user-agent, then add one Allow or Disallow path per line.

  3. 03
    Add sitemap URLs

    Enter any complete sitemap or sitemap-index URLs. The preview and checks update as you type.

  4. 04
    Copy or download robots.txt

    Review full-site blocking warnings, then copy the output or download the ready-to-upload robots.txt file.

03

Who it's for

  • Site owners creating a first robots.txt file without hand-writing every directive.
  • SEO specialists checking crawl rules before they are placed on a production domain.
  • Developers documenting crawler access for admin areas, search pages, or generated URL spaces.
  • WordPress administrators starting with a focused wp-admin rule and an exception for admin-ajax.php.

A robots.txt file belongs at the root of a site, such as https://example.com/robots.txt. This builder follows the core Robots Exclusion Protocol fields defined by RFC 9309 and adds the widely supported Sitemap field. It checks user-agent tokens, path formatting, and absolute sitemap URLs while you type. Presets give you a safe starting point for an open site, WordPress, or a complete crawl block.

Robots rules control crawling, not access or guaranteed indexing. Anyone can read the file, and crawlers can choose whether to honor it. Use authentication or server permissions for private content, and use a supported noindex method when the goal is to keep a page out of search results.

FAQ

Is this robots.txt generator free?

Yes. The generator is free to use with no account, sign-up, or usage limit. You can create multiple user-agent groups and sitemap entries, then copy the output or download it as robots.txt.

Is my robots.txt configuration uploaded?

No. The rules and sitemap URLs you enter are processed locally in your browser. The generator does not upload your configuration or send it to a server.

How many rules or sitemap URLs can I add?

The tool sets no fixed limit on groups, paths, or sitemap URLs. Search crawlers may impose their own file-size handling limits, so keep the final file focused and avoid duplicate rules.

Where should I upload robots.txt?

Place the UTF-8 plain-text file at the top-level /robots.txt path for the exact scheme and host it controls. For example, rules for https://www.example.com belong at https://www.example.com/robots.txt, not in a subdirectory.

Does robots.txt stop a page from appearing in Google?

Not reliably. A Disallow rule asks compliant crawlers not to request a URL, but Google may still know the URL from links and show it without a snippet. Use an appropriate noindex directive when you need to prevent indexing, and do not block the crawler from seeing that directive.

Should I add Crawl-delay?

Only when you know the target crawler supports it. Crawl-delay is not part of the core RFC 9309 rule syntax, and Google does not support it. This generator leaves it out so the main workflow stays portable and standards-aware.

Can robots.txt protect private files?

No. A robots.txt file is public and its paths can reveal where content lives. It is a voluntary crawling instruction, not authentication or authorization. Protect sensitive resources with login controls, server permissions, or another real access-control layer.