Which AI crawlers can fetch your homepage

The useful question is not "do we block AI." It is which product token may fetch /, and which line in the file decides it.

The robots.txt checker for AI crawlers fetched four hosts on 12 November 2026. The scores below are what it returned. A token counts in a score only when its operator says the file applies. ChatGPT-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher, Google-Agent, and Google-GeminiNotebook are outside those fractions. Their operators say the fetcher may ignore robots.txt.

What did four live files actually allow?

Two sites leave the homepage open to every token that honors the file. One names the bots and blocks most of them. One has no file.

The answer denominator is Claude-User and DuckAssistBot. The search denominator is OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot, Meta-WebIndexer, YouBot, and Diffbot. The training denominator is GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, and Meta-ExternalAgent. AI2Bot and Bytespider stay out of the fractions. Their operator pages do not say the crawler reads robots.txt.

Why does this site allow /, then block /api/?

aiagentsinfo.com's file, after one redirect to https://www.aiagentsinfo.com/robots.txt, gives each named token Allow: / and Disallow: /api/. The checker matched those groups. On /, answer fetches were 2/2, search 8/8, and training controls 6/6. On /api/, each of those scores was 0. The longer rule wins, so /api/ is blocked even though / is allowed.

The file has a Sitemap line. /llms.txt returned HTTP 200. That file is not a robots rule. Google-Extended being allowed does not change Google Search. Google says the token is a control for Gemini training and grounding, and that it has no separate HTTP user agent.

The checker's fix list, on this file, points at the split OpenAI publishes: GPTBot may train models, OAI-SearchBot is the search switch, and the two are independent. This site currently allows both. That is a choice, not an accident in the parser.

What does GitHub allow on /, and what does it block on /gist/?

GitHub's file puts GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User in one group. That group does not disallow /. The checker allowed all of them on the homepage, and it allowed the training controls that are not named, because User-agent: * does not disallow / either. /llms.txt returned HTTP 200. One Sitemap line was present.

The same GPTBot group does disallow /gist/. Checked on that path, GPTBot, OAI-SearchBot, Applebot, and Google-Extended were blocked. Bytespider was blocked on / and on /gist/, by Disallow: / in its own group. Bytespider is not in the score. Toutiao's page names the token and does not say it obeys robots.txt.

A shared group is the thing people miss. Those eight tokens follow one set of rules. Editing GPTBot means editing OAI-SearchBot, unless you split them.

Why is the Times score 0/2, 2/8, and 1/6?

The Times file names the bots and, for most of them, says Disallow: /.

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, YouBot, and Diffbot are each disallowed on /. That is 6 of the 8 search tokens. The two allowed search tokens are Applebot and Amzn-SearchBot. Neither is named. They follow User-agent: *, and that group does not disallow /. Apple's page says that if the file skips Applebot and names Googlebot, Applebot follows the Googlebot group. On this file those two agents share a group, and that group also allows /. The checker printed both facts.

Claude-User and DuckAssistBot are disallowed on /. That is the 0/2. Anthropic says a Claude-User block stops retrieval for a user question. ChatGPT-User is also Disallow: /, and OpenAI says robots.txt may not apply to it, so it is not in the 0/2.

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and Meta-ExternalAgent are disallowed on /. Amazonbot is the 1 of 6. Its group does not disallow /. It disallows /wirecutter/. On that path the checker blocked Amazonbot and still blocked GPTBot, because GPTBot's rule is the whole site.

/llms.txt returned HTTP 404. The file has 25 Sitemap lines. A missing llms.txt does not change the robots rules.

What happens when the file is missing?

example.com returned HTTP 404 for /robots.txt and for /llms.txt. The body was HTML. The checker did not parse that HTML as rules. It treated the status as a missing file. RFC 9309 then lets a crawler fetch public URLs, so every honoring token was allowed. There was no Sitemap line, because there was no file.

A 404 is not a polite block. It is the opposite. If you want training crawlers out, you have to publish a file that names them.

A 5xx would have gone the other way. RFC 9309 says an unreachable robots.txt is a complete disallow until a fetch works. This run did not hit a 5xx.

What should you publish if you want search in and training out?

Use separate groups. Allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot if you want those indexes. Disallow GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended if you want out of those training uses. Repeat any path you care about inside each allowed group. A named group does not inherit User-agent: *.

Do not expect the file to stop ChatGPT-User, Perplexity-User, or Google-Agent. The operator pages say those fetches may ignore it. robots.txt is a request. It is not access control, and an allow is not a citation.

The checker that produced this page is the robots.txt checker for AI crawlers. It can fetch another host, or take a paste, or write the split above. The host is not stored. The week of fetches behind the "who is even asking" question is the November 2026 answer-crawler report.