Submit your tool

Daily brief

Do AI agent companies let AI agents read their sites? robots.txt and llms.txt of 52 websites, November 2026

On 13 November 2026, 45 of 47 parsed robots.txt files allow / for GPTBot. Two full-block it. 27 of 52 websites returned an llms.txt file.

Checked November 13, 2026. robots.txt and llms.txt were fetched at 2026-11-13T03:21:43.733Z. Each homepage status is one GET from 54.147.134.223.

Published Updated 8 min readAI Agents Info Research

  • 45 of 47

    Parsed files that allow / for GPTBot

    Amazon and Onton disallow /. The other parsed files allow the path.

  • 1 of 47

    Parsed files that full-block ChatGPT-User

    Amazon sets Disallow: /. Claude-User matches that count.

  • 27 of 52

    Sites with llms.txt as HTTP 200 text

    HTML 200 does not count. OpenAI, claude.com, and Gemini are outside it.

  • 44 of 52

    Homepages that returned HTTP 200 to ChatGPT-User

    One GET from 54.147.134.223. Claude-User returned HTTP 200 on 45 of 52.

Executive summary

On 13 November 2026 this page records robots.txt and llms.txt for 52 websites that sell or publish an AI agent. 47 files parsed. Four hosts returned 404. ServiceNow returned 403. 45 of the 47 allow / for GPTBot. 113

The block that does exist is narrow and named. Amazon disallows / for the OpenAI, Anthropic, and Perplexity tokens in this set, and for Devin and Manus-User. Onton disallows / for GPTBot, Google-Extended, and Bytespider. GitHub disallows / for Bytespider and disallows /copilot/ for GPTBot and Claude-User. The lab marketing hosts in the 52 do not full-block one another. 123

Nothing here is estimated. A rule is copied from the file fetched at 2026-11-13T03:21:43.733Z. A homepage status is one request. A 4xx on robots.txt is unread. RFC 9309 says a client error means the crawler may still fetch. 13

Key findings

  1. 0145 of 47 robots.txt files that parsed on 13 November 2026 allow the path / for GPTBot. Amazon logoAmazon and Onton are the two that name GPTBot and set Disallow: /. [[1]][[2]]
  2. 02Amazon logoAmazon is the only parsed file that full-blocks ChatGPT logoChatGPT-User, Claude logoClaude-User, and Perplexity logoPerplexity-User. Googlebot is full-blocked on 0 of the 47. On Amazon logoAmazon, Googlebot is not named, so the star group applies, and that group does not disallow /. [[1]]
  3. 0327 of 52 websites returned /llms.txt with HTTP 200 and a non-HTML body. Among OpenAI logoOpenAI, Anthropic logoAnthropic’s Claude logoclaude.com, Gemini logoGemini, Microsoft logoMicrosoft, MMeta, xAI logoxAI, Mistral, and Cohere, the text files were Mistral and Cohere. [[6]][[10]]
  4. 04https://Claude logoclaude.ai/robots.txt, fetched the same hour and kept out of the 52, names GPTBot, OAI-SearchBot, ChatGPT logoChatGPT-User, and Google logoGoogle-Extended and sets Disallow: /. Claude logoClaude-User is not named. The star group allows / and disallows /api/* and /lti/*. [[4]][[11]]
  5. 05One GET with OpenAI logoOpenAI’s published ChatGPT logoChatGPT-User agent returned HTTP 200 for 44 of 52 homepages. The Claude logoClaude-User token, which Anthropic logoAnthropic’s help page names and does not expand into a Mozilla string, returned HTTP 200 for 45 of 52. Both requests came from 54.147.134.223. OpenAI logoopenai.com allows / and returned 403. [[6]][[12]]

Who full-blocks a training crawler, and who full-blocks a user-fetch agent?

45 of 47 parsed files allow / for GPTBot. The chart counts the files that do the opposite: Disallow wins for /. Bytespider is the token with the most full blocks, at 3. ChatGPT-User and Claude-User each have 1. 13

Of the 47 parsed files. A longer bar is more full blocks of /. Googlebot is 0. Applebot-Extended is off this chart: Onton matches it through the Applebot prefix.
Parsed files that disallow / for the tokenOf the 47 parsed files. A longer bar is more full blocks of /. Googlebot is 0. Applebot-Extended is off this chart: Onton matches it through the Applebot prefix. Unit: Parsed files. robots.txt fetched 13 November 2026, 2026-11-13T03:21:43.733Z. [[1]][[2]][[3]]GPTBot2OAI-SearchBot1ChatGPT-User1ClaudeBot1Claude-User1Claude-SearchBot1PerplexityBot1Perplexity-User1Google-Extended2CCBot1Bytespider3meta-externalagent1
Parsed files that disallow / for the token
CategoryFull block of /
GPTBot2
OAI-SearchBot1
ChatGPT-User1
ClaudeBot1
Claude-User1
Claude-SearchBot1
PerplexityBot1
Perplexity-User1
Google-Extended2
CCBot1
Bytespider3
meta-externalagent1

robots.txt fetched 13 November 2026, 2026-11-13T03:21:43.733Z. 123 Unit: Parsed files.

Rule for / on the 47 parsed files. 404s and the ServiceNow 403 are excluded. Onton’s Applebot prefix is counted on Applebot-Extended. Source: each robots.txt URL below. 13 November 2026.
TokenFull block of /PartialAllowedNot namedNamed groupChecked
GPTBot227153132026-11-13
OAI-SearchBot12815382026-11-13
ChatGPT-User128153122026-11-13
ClaudeBot128153122026-11-13
Claude-User12815362026-11-13
Claude-SearchBot12815362026-11-13
PerplexityBot127163112026-11-13
Perplexity-User12815362026-11-13
Google-Extended226163102026-11-13
Googlebot02915332026-11-13
Applebot-Extended12716382026-11-13
CCBot12716362026-11-13
Bytespider32714352026-11-13
meta-externalagent12716342026-11-13

3 parsed files full-block GPTBot, ClaudeBot, CCBot, or Bytespider: Amazon, Onton, GitHub. 1 full-blocks ChatGPT-User, Claude-User, or Perplexity-User: Amazon. Onton and GitHub full-block a training token and still allow / for those three user-fetch tokens. 123

Amazon sets Disallow: / on GPTBot, ChatGPT-User, Claude-User, the other lab tokens in this set, Devin, and Manus-User. Googlebot is not named. The star group disallows cart and sign-in paths and allows /. Onton sets Disallow: / on GPTBot, Google-Extended, and Bytespider. Its star group allows /. Onton names Applebot, not Applebot-Extended. This matcher treats that as a prefix, so Applebot-Extended inherits Disallow: /. 12

GitHub sets Disallow: / on Bytespider. The group that names GPTBot and Claude-User allows / and disallows /copilot/. /features/copilot stays allowed. 3

Which files name a rival, and which leave the homepage open?

14 of 47 parsed files name at least one of the 14 tokens. OpenAI, claude.com, Gemini, Microsoft, Meta, and xAI name none of them, and none of those six full-blocks a rival. openai.com allows / and disallows only /microsoft-for-startups/. claude.com is Allow: /. Gemini disallows /chat/ and /app/ for the star group, so Googlebot, which is not named, inherits that. 5610

When individuals ask questions to Claude, it may access websites using a Claude-User agent.

Anthropic help center, read 13 November 2026. 11

The same page says Anthropic’s bots honor robots.txt. claude.ai, fetched the same day, disallows / for GPTBot, OAI-SearchBot, ChatGPT-User, and Google-Extended. Claude-User is on the star group, which allows / and disallows /api/*. claude.com allows / and names none of these tokens. The app host blocks the rival user-fetch token. The marketing host does not. 4511

Second hosts, fetched 13 November 2026, excluded from the 52. claude.ai is the file that full-blocks OpenAI’s tokens. chatgpt.com does not.
HostGPTBotChatGPT-UserClaude-UserClaudeBotGoogle-Extendedllms.txtChecked
chatgpt.comPartial*Partial*Partial*Partial*Full4042026-11-13
www.anthropic.comAllowed*Allowed*Allowed*Allowed*Allowed*4042026-11-13
claude.aiFullFullPartial*Partial*FullHTML2026-11-13
www.google.comPartial*Partial*Partial*Partial*Partial*4042026-11-13
cognition.aiPartial*Partial*Partial*Partial*Partial*Text2026-11-13
copilot.microsoft.comPartial*Partial*Partial*Partial*Partial*HTML2026-11-13

Cursor’s star group allows / and disallows /agents. GPTBot is not named, so /agents is closed to it. devin.ai allows / for GPTBot and disallows /top-secret. Browser Use writes Allow: / for GPTBot, ChatGPT-User, and Claude-User. Amazon’s Disallow: / for the token Devin is the other side of that file. 1789

Who returned an llms.txt file on November 13?

27 of 52 requests returned HTTP 200 with a non-HTML body. 20 were 404, 3 were 403, and 2 were HTML, including v0.dev and bolt.new after redirects. Among OpenAI, claude.com, Gemini, Microsoft, Meta, xAI, Mistral, and Cohere, the text files were Mistral and Cohere. adk.dev returned text. gemini.google.com returned 404. 610

Cursor, Devin, CrewAI, Replit, Lovable, GitHub, Salesforce, and Browser Use are in the 27. Salesforce’s final URL carried ?bc=DB and the type was text/plain. Cartesia and Salesforce hit the 64 KB cap after a Markdown first line, so they stay in. Aidelly, adk.dev, and a2ui.org have no robots.txt and do have llms.txt. Continue has neither. 313

What did one request from one cloud IP get back?

One GET each, from 54.147.134.223, at 2026-11-13T03:21:43.733Z. ChatGPT-User used the Mozilla string on OpenAI’s crawler page. Claude-User was the token alone. Anthropic’s page names that token and prints no Mozilla string. 1112

Homepage status from one GET each. Challenge means the first 32 KB contained one of these markers: just a moment, cf-browser-verification, cf-challenge, attention required, captcha, access denied, enable javascript. Checked 13 November 2026.
ResultChatGPT-UserClaude-UserChecked
HTTP 20044452026-11-13
HTTP 200 that also contained a challenge marker112026-11-13
HTTP 403642026-11-13
HTTP 503112026-11-13
No status (fetch failed)122026-11-13

Browserbase returned 200 with a captcha marker for both. OpenAI returned 403 with no marker for both. ChatGPT-User also got a challenge 403 from Perplexity, Lovable, Salesforce, xAI, and ServiceNow. Claude-User got that 403 from Perplexity, Lovable, and xAI. Salesforce returned 200 to Claude-User. Amazon returned 503 to both. Gemini failed before a status. 6

Because these actions are initiated by a user, robots.txt rules may not apply.

OpenAI, Overview of OpenAI Crawlers, the ChatGPT-User row, read 13 November 2026. 12

openai.com allows / and the homepage returned 403 to ChatGPT-User. The file and the response are two measurements. One address, one try. 612

What does each of the 52 files do to these 14 tokens?

Full means Disallow wins for /. Partial means / is allowed and another path is disallowed. A star means the star group applied. One label means all 14 tokens share it. Text means llms.txt was HTTP 200 and not HTML. Factory resolved to factory.com. Checked 13 November 2026.

All 52 websites. The robots.txt link is the file that was parsed. Every row was checked 13 November 2026.
Vendorrobots.txtRule for the 14 tokensllms.txtChecked
AG2www.ag2.ai/robots.txtPartial*4042026-11-13
Aidellywww.aidelly.ai/robots.txtNo fileText2026-11-13
Aideraider.chat/robots.txtNot named4042026-11-13
Amazonwww.amazon.com/robots.txtPartial*: Googlebot, Applebot-Extended. Full × 124042026-11-13
Augment Codewww.augmentcode.com/robots.txtAllowed*Text2026-11-13
Browser Usebrowser-use.com/robots.txtPartial*: Bytespider. Allowed × 13Text2026-11-13
Anthropicclaude.com/robots.txtAllowed*4042026-11-13
Clinecline.bot/robots.txtPartial: GPTBot, ChatGPT-User, ClaudeBot. Allowed* × 11Text2026-11-13
CNVScnvs.dev/robots.txtNot named4042026-11-13
OpenAIopenai.com/robots.txtPartial*4032026-11-13
CrewAIcrewai.com/robots.txtPartial*Text2026-11-13
Cursorcursor.com/robots.txtPartial*Text2026-11-13
Cognitiondevin.ai/robots.txtPartial* × 6. Partial × 8Text2026-11-13
Google ADKadk.dev/robots.txtNo fileText2026-11-13
LangChainwww.langchain.com/robots.txtNot named4042026-11-13
Marblismwww.marblism.com/robots.txtPartial*Text2026-11-13
NoMacnomac.app/robots.txtPartial*Text2026-11-13
Ontononton.com/robots.txtFull, via Applebot: Applebot-Extended. Full: GPTBot, Google-Extended, Bytespider. Partial* × 104042026-11-13
Perplexitywww.perplexity.ai/robots.txtPartial: Googlebot. Partial* × 134032026-11-13
Pydanticpydantic.dev/robots.txtAllowed × 7. Allowed* × 7Text2026-11-13
Sintrasintra.ai/robots.txtPartial*4042026-11-13
A2UIa2ui.org/robots.txtNo fileText2026-11-13
SiteGPTsitegpt.ai/robots.txtPartial*: Googlebot, CCBot, Bytespider. Partial × 11Text2026-11-13
Hugging Facehuggingface.co/robots.txtAllowed*4042026-11-13
Cartesiawww.cartesia.ai/robots.txtAllowed*Text2026-11-13
ElevenLabselevenlabs.io/robots.txtPartial*Text2026-11-13
1ClickWebsitewww.1clickwebsite.ai/robots.txtAllowed*4042026-11-13
Googlegemini.google.com/robots.txtPartial*4042026-11-13
Microsoftwww.microsoft.com/robots.txtPartial*4042026-11-13
Replitreplit.com/robots.txtAllowed*Text2026-11-13
Lovablelovable.dev/robots.txtPartial* × 6. Partial × 8Text2026-11-13
Salesforcewww.salesforce.com/robots.txtPartial*Text2026-11-13
GitHubgithub.com/robots.txtFull: Bytespider. Partial* × 5. Partial × 8Text2026-11-13
AWSaws.amazon.com/robots.txtPartial*4042026-11-13
Mistralmistral.ai/robots.txtAllowed*Text2026-11-13
Coherecohere.com/robots.txtPartial*Text2026-11-13
xAIx.ai/robots.txtPartial*4042026-11-13
Metaai.meta.com/robots.txtPartial*4042026-11-13
Manusmanus.im/robots.txtPartial*4042026-11-13
Vercelv0.dev/robots.txtAllowed*HTML2026-11-13
StackBlitzbolt.new/robots.txtPartial*HTML2026-11-13
Continuecontinue.dev/robots.txtNo file4042026-11-13
Sourcegraphampcode.com/robots.txtPartial*Text2026-11-13
Factoryfactory.com/robots.txtPartial* × 5. Partial × 9Text2026-11-13
Warpwww.warp.dev/robots.txtPartial* × 6. Allowed × 8Text2026-11-13
Browserbasewww.browserbase.com/robots.txtAllowed*Text2026-11-13
Sierrasierra.ai/robots.txtAllowed*4042026-11-13
Gleanwww.glean.com/robots.txtPartial × 6. Partial* × 84042026-11-13
ServiceNowwww.servicenow.com/robots.txtNot read4032026-11-13
Workdaywww.workday.com/robots.txtPartial*4042026-11-13
LlamaIndexwww.llamaindex.ai/robots.txtAllowed*: Googlebot, Bytespider, meta-externalagent. Allowed × 11Text2026-11-13
Mastramastra.ai/robots.txtAllowed*Text2026-11-13

What was left out?

The 52 hosts were fixed before the first request. No connection failed, so no reserve was added. Docs-only URLs on GitHub, Hugging Face, learn.microsoft.com, developers.openai.com, and code.claude.com were skipped. GitHub and Hugging Face stay because the listing URL is the company host. Google, Microsoft, and Amazon each contribute two hosts: adk.dev and gemini.google.com, github.com and microsoft.com, aws.amazon.com and amazon.com.

This is not the 12 November guide that fetched four hosts, and it is not this site’s DataFast log. github.com is in both. These lines are the 13 November fetch. A disallow past the probed paths can still exist. The homepage test did not retry. CCBot, Bytespider, Perplexity, Google-Extended, Applebot-Extended, and meta-externalagent were not re-documented today. A full block is still a line in the file.

FAQ

Do these 52 companies let an agent read the homepage?

45 of 47 parsed files allow / for GPTBot. Amazon and Onton disallow it. 46 of 47 allow / for ChatGPT-User. Amazon disallows it. Four hosts had no file. ServiceNow returned 403.

Does Amazon block Googlebot?

Amazon does not name Googlebot. The star group allows /. GPTBot, ChatGPT-User, and Claude-User are named, each with Disallow: /.

Why do claude.com and claude.ai disagree?

Both files were HTTP 200 on 13 November 2026. claude.com allows / and names none of these tokens. claude.ai disallows / for GPTBot, OAI-SearchBot, ChatGPT-User, and Google-Extended. Claude-User stays on the star group, which allows /.

Does an llms.txt file mean the crawler is allowed?

27 of 52 hosts returned text at /llms.txt. That URL is separate from robots.txt. OpenAI, claude.com, and gemini.google.com allow / for GPTBot and returned no text file.

Did the homepage test prove that a bot is blocked?

It recorded one status from 54.147.134.223. openai.com allows / and returned 403 to ChatGPT-User and to Claude-User. OpenAI says robots.txt rules may not apply to ChatGPT-User. A 403 from one cloud IP is a status.

Methodology

The 52 hosts were listed before the first request: directory agent, coding-agent, shopping-agent, and ai-agent sites on the vendor’s own domain, plus SiteGPT, Cartesia, ElevenLabs, 1ClickWebsite, and A2UI, plus Google, Microsoft, Replit, Lovable, and Salesforce, plus AWS, Mistral, Cohere, xAI, Meta, Manus, v0.dev, bolt.new, Continue, ampcode.com, Factory, Warp, Browserbase, Sierra, Glean, ServiceNow, Workday, LlamaIndex, and Mastra. Every primary connected. chatgpt.com, www.anthropic.com, claude.ai, www.google.com, cognition.ai, and copilot.microsoft.com are second hosts, outside the 52.

GET aiagentsinfo-vendor-robots-study/1.0 (+https://www.aiagentsinfo.com/reports/ai-agent-vendor-robots-txt-november-2026). Parse only HTTP 200 text. The matcher is case-insensitive, longest prefix, then the star group, longest path, Allow on a tie. Full block means Disallow wins for /. Partial means / is allowed and a probed disallow still blocks. Finished 2026-11-13T03:21:43.733Z. Homepage: one GET per agent from 54.147.134.223, 20 seconds, 32 KB. llms.txt capped at 64 KB. RFC 9309 says a missing group must obey the star group, no group and no star means no rules, HTTP 400–499 may be fetched, and HTTP 500–599 must be treated as disallow. This sample had no 5xx on robots.txt. 13

Sources

  1. 1Amazon robots.txt. Fetched 13 November 2026 at 2026-11-13T03:21:43.733Z. HTTP 200, 7,947 bytes. GPTBot, ChatGPT-User, Claude-User, Devin, and Manus-User are Disallow: /. Googlebot is not named.
  2. 2Onton robots.txt. Fetched 13 November 2026. HTTP 200. GPTBot, Google-Extended, and Bytespider are Disallow: /. The Applebot group is Disallow: /. The star group allows /.
  3. 3GitHub robots.txt. Fetched 13 November 2026. HTTP 200. Bytespider is Disallow: /. The group that names GPTBot and Claude-User disallows /copilot/ and allows /.
  4. 4claude.ai robots.txt. Fetched 13 November 2026. HTTP 200, 281 bytes. Not one of the 52. GPTBot, OAI-SearchBot, ChatGPT-User, and Google-Extended are Disallow: /. The star group allows /.
  5. 5claude.com robots.txt. Fetched 13 November 2026. HTTP 200. One group, User-agent: *, Allow: /. This is the Anthropic row in the 52.
  6. 6openai.com robots.txt. Fetched 13 November 2026. HTTP 200. User-agent: *, Allow: /, Disallow: /microsoft-for-startups/. The homepage GET returned 403. /llms.txt returned 403.
  7. 7Cursor robots.txt. Fetched 13 November 2026. HTTP 200. User-agent: * allows / and disallows /agents and /*/agents.
  8. 8devin.ai robots.txt. Fetched 13 November 2026. HTTP 200. The named lab tokens and the star group allow / and disallow /top-secret and /assets/downloads/.
  9. 9Browser Use robots.txt. Fetched 13 November 2026. HTTP 200. Named groups for GPTBot, ChatGPT-User, ClaudeBot, and Claude-User are Allow: /. The star group allows / and disallows /api/.
  10. 10gemini.google.com robots.txt. Fetched 13 November 2026. HTTP 200, 116 bytes. User-agent: * allows /app/download and disallows /app/ and /chat/. /llms.txt returned 404. adk.dev/llms.txt returned 200 text.
  11. 11Anthropic, Does Anthropic crawl data from the web?. Read 13 November 2026. Claude-User “may access websites” when a person asks a question. The page says the bots honor robots.txt. It names the token Claude-User and does not print a Mozilla user-agent string.
  12. 12OpenAI, Overview of OpenAI Crawlers. Read 13 November 2026. ChatGPT-User user agent copied from that page. The page says robots.txt rules may not apply because a person started the fetch. GPTBot is the training crawler.
  13. 13RFC 9309, Robots Exclusion Protocol. Read 13 November 2026. Sections 2.2.1 and 2.3.1. No matching group must obey the star group. No group and no star group means no rules apply. HTTP 400–499 may be fetched. HTTP 500–599 must be treated as a complete disallow.

Cite or embed

The citation names this page and the date above. Every count on the page is computed from the robots.txt and llms.txt responses fetched that hour.

Citation

AI Agents Info. "Do AI agent companies let AI agents read their sites? robots.txt and llms.txt of 52 websites, November 2026." Last updated November 13, 2026. https://www.aiagentsinfo.com/reports/ai-agent-vendor-robots-txt-november-2026.

Embed

<blockquote cite="https://www.aiagentsinfo.com/reports/ai-agent-vendor-robots-txt-november-2026">
<p>Checked November 13, 2026. 45 of 47 parsed robots.txt files allow / for GPTBot. Amazon and Onton full-block it. 27 of 52 websites returned llms.txt as text. One homepage request per bot, from one cloud IP.</p>
<footer><a href="https://www.aiagentsinfo.com/reports/ai-agent-vendor-robots-txt-november-2026">Do AI agent companies let AI agents read their sites? robots.txt and llms.txt of 52 websites, November 2026</a>, AI Agents Info, last updated November 13, 2026.</footer>
</blockquote>

Related reading