Skip to content
ToolMug

Use ↑ ↓ to move, Enter to open, Esc to close.

Robots.txt Generator & Tester

Generate: choose the paths to block, the crawlers to keep out and your sitemap URLs, and get a robots.txt ready to upload. Test: fetch any site's robots.txt (or paste one), enter a URL and a crawler, and see whether it may crawl and which line decided it.

Processed on our server; nothing is stored

One per line, starting with /. Use * as a wildcard and $ for the end of the URL.

Exceptions inside blocked folders, e.g. /admin/public/

Optional. Bing and Yandex use it; Google and Baidu ignore it.

Block AI crawlers

Names published by each company. They only work for crawlers that respect robots.txt.

Block specific search engines

robots.txt

Upload the file to the root of the site so it is served at https://your-domain/robots.txt.

How to use

  1. In Generate, list the paths to block, tick any AI crawlers or search engines to block and add your sitemap URLs. The robots.txt updates as you go; copy or download it.
  2. In Test, enter a website and click Fetch robots.txt, or paste the content.
  3. Enter the URL to test and pick a crawler. The result says whether it may crawl and highlights the matching line.

Options explained

Matching rules
As in Google's specification — the group that best matches the crawler name applies, the longest matching path wins, Allow wins a tie, and * and $ are supported.
Crawl-delay
Seconds between requests. Bing and others honour it; Google and Baidu ignore it.

A worked example

Here is a robots.txt that blocks a folder but opens one subfolder, blocks PDF files, and keeps one AI crawler out entirely:

User-agent: *
Disallow: /private/
Allow: /private/press/
Disallow: /*.pdf$

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

The Test tab gives these answers:

Crawler URL path Result Deciding line
Googlebot /blog/post Allowed No rule matches
Googlebot /private/data Blocked Disallow: /private/
Googlebot /private/press/kit Allowed Allow: /private/press/ (longer match wins)
Googlebot /files/a.pdf Blocked Disallow: /*.pdf$
Googlebot /files/a.pdf?x=1 Allowed The $ means the URL must end there
GPTBot /blog/post Blocked Disallow: / in its own group
Googlebot /robots.txt Allowed The file itself is always crawlable

Googlebot uses the * group because no group names it, while GPTBot uses its own group and ignores the * rules. Matching follows RFC 9309, which defines only User-agent, Allow and Disallow. Sitemap and Crawl-delay are widely used extensions that the RFC does not define.

Frequently asked questions

Does robots.txt keep a page out of search results?

Not fully. It only asks crawlers not to fetch the page; if other sites link to it, the URL can still be indexed. Use a noindex tag for pages that must stay out, and do not block them in robots.txt, or crawlers never see the noindex.

Does blocking AI crawlers work?

Official crawlers such as GPTBot, ClaudeBot and Google-Extended say they respect robots.txt, so blocking them stops your pages being used for training. Crawlers that ignore the standard cannot be stopped this way.

Whose connection fetches robots.txt?

Our server requests the robots.txt at the site's root — that single file only. Testing pasted content happens entirely in your browser.

Privacy

This tool needs our server: what you enter or upload is sent over HTTPS, processed and then discarded, never stored. Server access logs record IP addresses and request URLs, used only to prevent abuse.

Related tools