AI Crawler Robots.txt Study

Key takeaways

1. Of 579 working homepages among the 1,000 most-visited domains, 110 (19%) block at least one AI crawler by name in robots.txt. See details
2. The most blocked are ByteDance's Bytespider (18%) and Common Crawl's CCBot (18%), then OpenAI's GPTBot (16%) and Anthropic's ClaudeBot (15%). Googlebot is blocked on 1.4%. See details
3. Sites block AI training more than AI search: GPTBot is kept off 16% of homepages but OpenAI's search crawler off 8%; ClaudeBot 15% against Claude-SearchBot 9%. See details
4. Google-Extended, the token that keeps a site's pages out of Gemini training and grounding, is blocked on 14%, about 10 times as often as Googlebot itself. See details
5. Sites pick and choose: The Guardian blocks Anthropic's crawlers but not GPTBot; CNN, Amazon, Medium and BBC block both; Wikipedia and Apple block neither, and the file Reddit served us blocks every crawler. See details
6. 106 homepages (18%) publish an llms.txt, among them GitHub, Samsung, Roku and Business Insider; 3 of them block GPTBot at the same time. See details

How many top websites block AI crawlers?

About one in five. We fetched the robots.txt file of each of the 579 working homepages among the 1,000 most-visited domains on the Tranco list and checked, for 19 AI crawler tokens and two search crawlers, whether the file lets that crawler fetch the homepage. 110 sites (19%) block at least one AI crawler by name. Separately, 26 files block every crawler through the catch-all "*" group; 18 of those still let Googlebot in by name. The test is the homepage, and some files open little else: 14 more let GPTBot fetch the homepage but keep it off the rest of the site, bar a few listed pages (an "Allow: /$" above "Disallow: /"), SoundCloud, Statista, Naver and Vimeo among them. Counting those, GPTBot is kept off everything beyond the homepage on 104 sites (18%), ClaudeBot on 103 (18%) and Googlebot on 16 (2.8%).

Which AI crawlers do websites block?

The most blocked are bulk crawlers: Bytespider on 102 homepages (18%), CCBot on 102, GPTBot on 90 and ClaudeBot on 89, the last three documented as gathering data for datasets and AI training. Google-Extended, a token Google documents for keeping pages out of Gemini training and grounding without affecting Search (it isn't a crawler of its own), is blocked on 81 (14%). Googlebot itself is blocked on 8 (1.4%), all of them sites that block every crawler.

Do sites block AI training or AI search?

Training, more than search. OpenAI and Anthropic each run separate crawlers for training, for search and for fetching a page a user asked about (OpenAI, Anthropic). GPTBot, OpenAI's training crawler, is blocked on 16% of homepages, but OAI-SearchBot on 8% and ChatGPT-User on 10%. Anthropic's split is the same: ClaudeBot 15%, Claude-SearchBot 9%, Claude-User 10%. 42 sites block GPTBot but let OAI-SearchBot in, and 36 block ClaudeBot but let Claude-SearchBot in: they keep their content out of training while staying open to AI search. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers".

What do the biggest websites do?

They don't agree. The Guardian blocks Anthropic's crawlers but not GPTBot; CNN, Amazon, Medium and BBC block both GPTBot and ClaudeBot; Wikipedia and Apple block neither. The robots.txt Reddit served our crawler blocks every crawler with a single "Disallow: /" for all user agents. Every site's verdict for every crawler is in the CSV.

How many websites publish an llms.txt?

106 homepages (18%) serve a real llms.txt, the plain-text Markdown summary the llmstxt.org proposal suggests sites publish for language models. They include GitHub, Samsung, Roku, Business Insider, Intercom and Dynatrace; the median file is 13 KB. Another 88 sites answered /llms.txt with an HTML page or a file that wasn't one, and aren't counted. 3 sites publish an llms.txt while blocking GPTBot.

What this means for your website

• Decide training and search separately. 42 of the 90 top sites that block GPTBot still let OpenAI's search crawler in.
• Name the crawler. Each of the main AI companies documents its own robots.txt token; a rule for "*" also blocks search engines.
• Check what you already block. Blocking OAI-SearchBot keeps a site out of ChatGPT search answers, by OpenAI's own account, and Anthropic says blocking Claude-User may reduce a site's visibility in Claude's web search.
• An llms.txt is cheap to add. 18% of top sites have one; it is a proposal, not a standard.

Mark Studios handles website SEO, including how a site appears in AI search, and builds websites.

How we ran the study

Sites. The 579 working homepages found among the 1,000 highest-ranked domains on the Tranco list ZJ8PG in our homepage video study, a list built for research (Le Pochat et al., NDSS 2019). 513 served a readable robots.txt. Of the other 66, 21 had none and 19 refused our crawler with a 403, which RFC 9309 treats as allowing every crawler; 19 sent an HTML page or a bot check, 3 redirected without reaching a file, and 4 gave a server error, a rate limit or no answer. For server and network errors RFC 9309 tells crawlers to assume a full block until the file can be read; we count all 66 sites as setting no rules, so the block counts are floors.

Files. On October 29, 2026 we fetched /robots.txt and /llms.txt from each site's homepage address with a crawler that named itself and linked to this research. A crawler counts as blocked when the file keeps it off the homepage ("/"), read the way RFC 9309 and Google describe: the group naming the crawler applies, otherwise the "*" group, and among matching rules the longest wins, with a tie going to Allow. Partial blocks that leave the homepage open aren't counted. An llms.txt counts when the address returns a file that isn't HTML and starts with a Markdown heading.

Checking the reading. Python's standard robots.txt parser agreed with our verdicts on 10,251 of 10,773 site-and-crawler pairs (95.2%). It applies the first matching rule instead of the longest and ignores the * and $ wildcards, so it differs on files such as google.com's and on sites that allow "/$" after "Disallow: /"; we follow the standard there. We also read a sample of the files and llms.txt bodies by hand. None of the sites or crawler operators reviewed or endorsed this study.

Part of a series. The same top-1,000 domains feed six studies, each from its own visit: homepage video, social media links, homepage SEO, third-party tools, AI crawlers in robots.txt and fonts.

Limitations

• Robots.txt is a request. It shows what a site asks, not what crawlers do or whether they comply.
• The homepage only. A site that blocks a crawler from some sections but not the homepage counts as not blocking it, including the 14 files that leave GPTBot only the homepage and a few listed pages.
• Files can differ by crawler. We read each file as our crawler received it; a site can send a different file to a crawler it recognizes.
• Some fetchers may not follow robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page (OpenAI, Perplexity).
• Tokens we checked. 19 AI crawler tokens. Three aren't crawlers of their own: Google-Extended and Applebot-Extended only tell Google and Apple how they may use pages their other crawlers fetch (Google, Apple), and anthropic-ai is an older token missing from Anthropic's current list. ByteDance and Cohere don't document Bytespider or cohere-ai. Sites may block other crawlers, or block by IP address or firewall instead.
• Domains, not companies. One company can appear several times, and some domains are services rather than publishers.

How to cite this study

Link to the section you use, for example https://www.markstudios.com/research/ai-crawler-robots-txt-study#training-vs-search. To cite the study as a whole:

Mark Studios. "AI Crawler Robots.txt Study." Published Oct 29, 2026. https://www.markstudios.com/research/ai-crawler-robots-txt-study

@misc{markstudios_ai_crawler_robots_txt_study_2026,
  author = {{Mark Studios}},
  title = {{AI Crawler Robots.txt Study}},
  year = {2026},
  month = oct,
  url = {https://www.markstudios.com/research/ai-crawler-robots-txt-study},
  note = {Last verified 2026-10-29}
}

Our measurements and the CSV are free to reuse under CC BY 4.0 with a link to this page.

Sources

Download the per-site results (CSV)

Widely quoted video statistics on other topics are traced to their origin, with a verdict on each, in our research fact-check.

1. Tranco, Tranco list ZJ8PG (generated October 28, 2026; ranks aggregated from Chrome UX Report, Farsight, Majestic, Cloudflare Radar and Cisco Umbrella, September 29 to October 28, 2026). Accessed 2026-10-29.
2. NDSS Symposium, Le Pochat et al., Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation (NDSS 2019). Accessed 2026-10-29.
3. IETF, RFC 9309: Robots Exclusion Protocol (2022). Accessed 2026-10-29.
4. Google Search Central, How Google interprets the robots.txt specification. Accessed 2026-10-29.
5. OpenAI, Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User). Accessed 2026-10-29.
6. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-User, Claude-SearchBot). Accessed 2026-10-29.
7. Google Search Central, Google's common crawlers (Google-Extended). Accessed 2026-10-29.
8. Perplexity, Perplexity Crawlers (PerplexityBot, Perplexity-User). Accessed 2026-10-29.
9. Apple, About Applebot (Applebot-Extended). Accessed 2026-10-29.
10. llmstxt.org, The /llms.txt file (proposal). Accessed 2026-10-29.

Update policy and change log

We plan to rerun the check every three months; AI crawler lists change fast. Corrections go to contact@markstudios.com.

• Oct 29, 2026: First published: robots.txt and llms.txt on 579 homepages among the 1,000 most-visited domains.