AI answer crawlers, November 2026: 196 fetches in 7 days, 160 from ChatGPT-User

Which answer engines fetched a page?

DataFast classified 196 requests as answer fetches. The field is aiAnswerRequests. The export used category "all" and verification "all". All 17,602 requests are IP-verified. User-agent-only requests are 0. ¹

Four user agents make up the 196. Each share is computed (160 ÷ 196 is 81.6 percent). After rounding to one decimal, the four shares sum to 100.0 percent.

Perplexity and Microsoft recorded 0 answer fetches. PerplexityBot still indexed 1,217 times. Bingbot indexed 88 times. ¹

What did ChatGPT-User open?

ChatGPT-User's 160 fetches hit 37 path strings. The homepage is 69, 43.1 percent (69 ÷ 160). ¹

Trust and legal pages, defined here as /verified-revenue, /terms, /editorial-policy, /privacy, and /about, were fetched 11, 9, 8, 4, and 1 times. The sum is 33: 20.6 percent of ChatGPT-User (33 ÷ 160) and 16.8 percent of all answer fetches (33 ÷ 196). Both percents are computed. Homepage plus those five paths is 102, 63.8 percent of ChatGPT-User (102 ÷ 160), computed. /submit was fetched 11 times and is not in that trust set.

Report articles, /reports/ paths other than the index and two legacy /pdf URLs, were fetched 22 times: the coding brief 8, the harness benchmark 5, the framework table 2, the shopping outlook 2, the 12-month outlook 2, and one each for the six-month, browser, and MCP outlooks. 22 ÷ 160 is 13.8 percent, computed. Two more fetches hit legacy /pdf URLs, one each on the harness benchmark and the 12-month outlook. They are counted and not linked.

Twenty-four other ChatGPT-User paths were fetched once, including /about, /directory/traxy, /directory/everygen-ai, both /pdf URLs, and one evaluation-tools URL with utm_source=chatgpt.com.

Did Google, Claude, and DuckDuckGo open the same pages?

They did not. GoogleAgent used two paths: 14 fetches of the framework table and 2 of /best/ai-agent-frameworks. 14 ÷ 16 is 87.5 percent, computed. Google's fetchers page, updated 2026-08-19 UTC, documents Google-Agent, with a hyphen, as a user-triggered agent fetcher that generally ignores robots.txt. It does not document the spelling GoogleAgent. The export has DataFast's label, not the raw User-Agent, so these 16 rows stay under that label. ¹⁶

Claude-User used three paths: /robots.txt 6 times (6 ÷ 12 is 50 percent), the shopping outlook 5 times, and the OpenClaw stars note once. Anthropic says Claude-User fetches on a user question, honors robots.txt, and that a block stops retrieval. The help page said "Updated this week" and dates a training signal to 30 October 2026. ¹⁴

DuckAssistBot used three paths: /robots.txt 5 times (5 ÷ 8 is 62.5 percent), the SpecCompact note twice, and /research once. DuckDuckGo says it crawls in real time for cited answers, does not train models, and applies a Disallow after 72 hours without changing organic rank. Treating any of this as a habit beyond this week is an inference. ¹⁸

What were the other 17,406 fetches for?

Answer fetches are 1.1 percent of 17,602 requests (196 ÷ 17602, computed). Indexing is 7,843. Training is 9,563. Company totals, each fully IP-verified: Google 9,331, OpenAI 6,070, Perplexity 1,217, Anthropic 800, Microsoft 88, Apple 44, Meta 44, DuckDuckGo 8. ¹

Computed from the copied totals: PerplexityBot is 15.5 percent of indexing, OAI-SearchBot 75.3 percent, and GoogleOther 90.8 percent of training. OpenAI says OAI-SearchBot and GPTBot are independent in robots.txt, and that a search change can take about 24 hours. OAI-AdsBot is documented and absent. Apple's page, published 4 September 2026, calls Applebot a search crawler and says the data may also train models. DataFast filed the 44 Applebot requests as training. ³⁵¹⁰

Which days carried the 196 answer fetches?

Hourly buckets are sitewide and name no engine, so there is no per-engine day series. 167 buckets are present. The missing hour is 6 November, 00:00 IST, and the present buckets already sum to the window totals. Answer fetches by date are 0, 0, 5, 17, 35, 63, 76. 10 November is 38.8 percent of the week (76 ÷ 196, computed). First answer hour: 6 November, 10:00 IST, 3 fetches. Busiest: 10 November, 22:00 IST, 12 fetches. ¹

How many humans arrived, and what does robots.txt allow?

The human brief prints "2026-11-04 to 2026-11-11 IST (7 full days + today so far)." Generated at 6:46 a.m. IST, it marks 11 November partial: 7 visitors and 16 pageviews. The bot window ends at the close of 10 November. Referrers: ChatGPT 4, Google 6, Bing 2, Direct/None 97, X 12, and one each from Medium, substack.com, intercom.com, and Product Hunt. No Perplexity, Claude, or DuckDuckGo row is printed. The 6 Google visitors are search referrals, separate from the 16 GoogleAgent fetches. The 2 Bing visitors are separate from Bingbot's 88 indexing fetches. ¹²

The sitemap on 11 November 2026 had 292 URLs. 35 were answer-fetched at least once (35 ÷ 292 is 12.0 percent, computed), with the query string stripped. Outside it: /robots.txt (11 fetches), two /pdf URLs, and /directory/everygen-ai. This report was not among the 292. ¹³

The served robots.txt repeats Allow: / and Disallow: /api/ for * and for 24 named agents: Googlebot, Google-Extended, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Applebot, Applebot-Extended, CCBot, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, DuckAssistBot, cohere-ai, Bytespider, YouBot, Diffbot, and AI2Bot. GoogleOther, Google-InspectionTool, and Google-Agent are not named, so * covers them. OpenAI, Google, and Perplexity say user-triggered fetchers may ignore the file. Anthropic says Claude-User honors it. A DuckDuckGo Disallow takes 72 hours. ^3467812

What was left out?

No per-engine day series and no raw User-Agent, so GoogleAgent stays separate from Google-Agent. No URL list for indexing or training: the pages payload is 39 answer-fetch paths and hasMore is false. Meta's crawler docs returned HTTP 500 on 11 November 2026, so meta-externalagent has no vendor purpose here. Counts stop at the end of 10 November. Search Console is not in the tables. The /pdf fetches and /directory/everygen-ai are not sitemap hits.

FAQ

Did an answer engine cite this site?

The log counts fetches. It does not store the answer. A request is not a citation.

Why is the ChatGPT figure 40 to 1?

Computed: 160 ChatGPT-User fetches in seven full days, divided by 4 referral visitors in a window that also includes part of 11 November. A fetch is a bot request. A visitor is a browser session. It is not a conversion rate.

Does Google document GoogleAgent?

Google documents Google-Agent, hyphenated, updated 2026-08-19 UTC. It does not document the spelling GoogleAgent. DataFast's label on the 16 fetches is GoogleAgent. The two strings are not merged.

Why did Claude-User fetch robots.txt six times?

The export records the path, not a reason. Anthropic says Claude-User honors robots.txt. Six of twelve fetches were that file, as were five of DuckAssistBot's eight.

Should a builder block GPTBot because it barely visited?

OpenAI treats GPTBot and OAI-SearchBot as independent. This file shows 7 and 5,903. This site allows both. This page does not recommend a change.