Submit your tool

Free tool

robots.txt checker for AI crawlers: who can fetch /

Enter a domain. See which documented AI bots may fetch a path, then download a robots.txt that matches the policy you picked.

Operator pages read . The worked example on four live hosts is Which AI crawlers can fetch your homepage.

Check a host, paste a file, or write one

The bot list was read from operator docs on 12 November 2026. A URL check fetches /robots.txt and /llms.txt and then drops the response. Nothing you type is stored.

How do you use the checker?

  1. Step 1

    Enter the host, not the page

    example.com and https://example.com/pricing both check that host's /robots.txt. The optional path is the URL you want a verdict for. The checker does not fetch that path.

  2. Step 2

    Read the score, then the group

    Answer fetches, search, and training controls are counted only when the operator says the file applies. Open a row to see which User-agent group won. A prefix match, such as Applebot winning for Applebot-Extended, is labeled.

  3. Step 3

    Paste a file if the host refuses the fetch

    A 401, 403, or 429 means this checker did not read the file. RFC 9309 lets a crawler treat a 4xx as allow-all. That is not an Allow line. Paste the file you already have. The paste stays in the browser.

  4. Step 4

    Write the file you meant

    The usual split is allow search and answer tokens, disallow training tokens, and repeat protected paths inside every allowed group. Copy or download. Regenerate switches between one shared group and one group per bot. The bytes are the same policy.

What does a three-group file actually do?

This fixture is parsed by the same function as the tool. The score is Answer fetches: 2/2 allowed. Search: 8/8 allowed. Training controls: 5/6 allowed. GPTBot is disallowed. OAI-SearchBot is allowed. ChatGPT-User's group blocks /, and OpenAI says that fetcher may ignore the file. * disallows only /api/, so a search bot with no group of its own may still fetch the homepage.

User-agent: *
Disallow: /api/

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Disallow: /

The trap is the named group. Once GPTBot has its own group, it stops reading *. Allowing / on the wildcard does not let GPTBot back in. The same is true in reverse: a Disallow: / on * does not block a bot whose own group says Allow: /.

Which tokens are on the list?

29 tokens, each with the page it was read from. A token is omitted when that page did not name it.

Left off on purpose

  • cohere-ai — Cohere's crawler page says Cohere does not currently crawl for foundation-model training, and the bot table is empty. The example token Coherebot is a hypothetical, so it is not scored. The string cohere-ai is not on that page. Source
  • anthropic-ai — Anthropic's crawler help page names ClaudeBot, Claude-User, and Claude-SearchBot. It does not name anthropic-ai. Source
  • FacebookExternalHit — Meta documents it as the link-preview crawler for Facebook, Instagram, and Messenger, and says it might bypass robots.txt for security checks. It is not an AI training or answer crawler, so it is not in the score. Source
  • Googlebot — Googlebot is the Search crawler. Google-Extended is the separate token for Gemini training and grounding. Google says Google-Extended does not affect inclusion in Google Search. Source
  • Google-NotebookLM — Google's user-triggered fetchers page lists Google-NotebookLM as a former Gemini Notebook agent, supported until August 2026. The current token on that page is Google-GeminiNotebook. Source

Other free tools

Next steps

Sources

Read , except where the page prints its own updated date.

Canonical URL: https://www.aiagentsinfo.com/tools/ai-crawler-robots-txt-checker

Questions people ask before they edit robots.txt

The answers follow the operator page or RFC 9309. A crawler that ignores the file will not be talked into obeying it.

What is a robots.txt checker for AI crawlers?

It fetches a host's /robots.txt, parses it under RFC 9309, and says whether each documented AI product token may fetch / and, if you add one, another path. It also writes a robots.txt from a policy you pick. It does not call a model.

Does blocking GPTBot keep a site out of ChatGPT search?

No. OpenAI's crawler page, read 12 November 2026, says GPTBot is the training crawler and OAI-SearchBot is the search crawler. The two robots.txt settings are independent. A site that opts out of OAI-SearchBot is not shown in ChatGPT search answers.

Does Google-Extended affect Google Search?

No. Google's common-crawlers page, last updated 2026-10-30 UTC and read 12 November 2026, says Google-Extended is a robots.txt token for Gemini training and grounding. It has no separate HTTP user agent, and it does not affect inclusion or ranking in Google Search.

What does a missing robots.txt mean?

HTTP 404 or 410 means the file is unavailable. RFC 9309 lets a crawler fetch any public URL. An empty file is the same result: no group matches, so nothing is disallowed. A 5xx is the opposite. RFC 9309 says the crawler must assume a complete disallow until the file can be fetched.

Do ChatGPT-User and Perplexity-User obey robots.txt?

OpenAI says ChatGPT-User rules may not apply, because a person started the fetch. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says Claude-User does honor it, and a block stops retrieval. This checker shows the file's rule and the operator's caveat on the same row.

Is the URL or the pasted file stored?

No. A URL check is sent to this site so the server can fetch /robots.txt and /llms.txt. The response is not written to a file, a database, or a log. Paste mode runs in the browser and is not sent. Do not paste secrets. robots.txt is public.

Why is a bot from my logs missing?

The list includes only product tokens this desk read on the operator's own documentation page on 12 November 2026. cohere-ai, anthropic-ai, FacebookExternalHit, Googlebot, and Google-NotebookLM are left out on purpose. The page says why for each one. A client can send any User-Agent, so a log line is not proof of the operator.

Will the generated file stop every AI bot?

No. robots.txt is a request, not access control. Tokens whose operators say they may ignore the file are named in the comments. A named group also replaces User-agent: *, so a path you want blocked has to be repeated inside that group.