The Open Index · Crawlers
AI bots take a lot, give back little, and don’t need a special file.
What Cloudflare’s network data shows about AI crawling, what the web-wide crawl and an independent study say about llms.txt, and why your content must be in the HTML.
38,066
requests made by Anthropic’s crawlers for each visit they referred
Cloudflare, July 2025
Written byKris GjorgjioskiKeyword research and industry benchmarks, Webtec Research
Checked byHristijan NajcheskiHead of SEO and AEO (technical and AI SEO), data verification
Updated October 15, 2026 · Edition 1Refreshed quarterly · next January 2027
Built from research on
Google
OpenAI
Perplexity
Gemini
Anthropic
Cloudflare
Semrush
TAKE AND GIVE
Most AI crawling is for training, which sends nobody back.
Cloudflare Radar: the purpose of AI crawling, and how many pages each platform crawls for every visit it refers.
Why AI bots crawl
Share of AI crawling by purpose, past 12 months.
- Training80%
- Search18%
- User actions2%
SourceCloudflare · 2025
Crawls per referred visit, July 2025
Bar length is on a log scale: Anthropic’s ratio is about 7,000 times Google’s.
- Anthropic38,065.7
- OpenAI1,091.4
- Perplexity194.8
- Microsoft40.7
- Yandex21.4
- Google5.4
SourceCloudflare · 2025
Cloudflare defines the ratio as pages crawled divided by visits referred. Anthropic’s ratio fell from 286,930 in January to 38,066 in July 2025; OpenAI’s stayed near 1,100 to 1,200.[1] Caveats: purpose labels rely on operators’ disclosures, and the referral data comes from Cloudflare customers that skew to news publishers.
THE FILE QUESTION
llms.txt: rare, and no measurable citation effect.
Two samples, two adoption rates, one finding.
Share of sites with llms.txt
Different samples give different rates. Both say the file is uncommon.
- SE Ranking sample of ~300,000 domains10.13%
- Web Almanac, desktop sites crawled2.13%
- Web Almanac, mobile sites crawled2.1%
WHAT WE KNOW
Optional, harmless, unproven.
SE Ranking’s statistical tests and machine-learning model found no effect of llms.txt on how often a domain is cited; the authors conclude it does not seem to impact citation frequency “at least not yet”.[4]
Google’s documentation says you do not need new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode.[5] The Web Almanac puts adoption at about 2% of crawled sites.[3]
THE ONE THAT MATTERS
Put your content in the HTML.
Vercel’s analysis of crawler traffic found that none of the major AI crawlers it studied render JavaScript. They fetch script files but do not run them, so content that appears only after scripts execute can be invisible to them.[2]
The check takes a minute: fetch a page as plain HTML and look for your own headline. We show how in check it yourself.
A SENSIBLE CHECKLIST
- Main text is in the server-sent HTML
- robots.txt allows the search and user-action crawlers you want
- Pages are indexable and eligible for snippets
- Key facts are stated in plain text, not only in images
- llms.txt is optional: publish if you like, do not rely on it
Industry context: how much traffic AI actually sends.[6]
OUR INTERPRETATION
What we make of the numbers.
The figures above are the studies’. This part is our reading of them, kept separate so you can disagree.
Webtec Research
Our interpretation
- Kris GjorgjioskiKeyword research and benchmarks
- Hristijan NajcheskiTechnical SEO, data checks
- Figures: the cited studies
- Reading: ours
The figures on this page come from the cited studies. This reading of them is ours, and you may weigh it differently.
Crawling and referring are far apart. AI companies fetch a great deal of content and send back few visitors, and the ratio differs by a factor of thousands between companies. That is a commercial fact, not a technical one, and it is why blocking decisions matter.
Our reading is that most businesses should distinguish between crawlers that find them (search) and crawlers that train on them (training). Allowing the first and choosing about the second is a defensible default. Files like llms.txt are cheap to add but have no measured effect on citation, so they should never replace the fundamentals: crawlable, fast, clearly written pages.
- Write a crawler policy by purpose, not a blanket block or allow
- Check your server logs for AI user agents quarterly
- Treat llms.txt as optional, not as a strategy
Cite this page.
Please link back and credit Webtec Research.
CITE THIS PAGE
Webtec Research. (2026, October 15). AI crawlers, robots.txt and llms.txt: what the data shows. The Open Index by Webtec. https://www.trywebtec.com/resources/index/ai-crawlers-and-files/
Questions about crawlers and files.
Does llms.txt help me get cited by AI?
The evidence says not measurably. SE Ranking analysed about 300,000 domains (published November 2025) and found no correlation between having llms.txt and AI citations; removing the variable even improved its model. Google’s documentation says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. It costs little to publish one, but do not expect a lift.
How much do AI bots crawl compared with what they send back?
A lot more. Cloudflare’s crawl-to-referral ratio for July 2025 was 38,066 for Anthropic, 1,091 for OpenAI, 195 for Perplexity, 41 for Microsoft and 5.4 for Google, meaning crawls per referred visit. Cloudflare classified 80% of AI crawling over twelve months as training, 18% as search and 2% as user actions.
Can AI crawlers read JavaScript-rendered pages?
Vercel’s 2024 research found that none of the major AI crawlers it studied render JavaScript. Put important content in the HTML the server sends, and test a page with a plain fetch to see what a crawler receives.
Should I block AI crawlers?
It is a business decision, not a technical default. Training crawlers take a lot and send back little, per Cloudflare. Search and user-action crawlers are what let assistants find and cite you. Google says access for its Search features is controlled through robots.txt, and Google-Extended controls some other uses.
What are AI crawlers actually doing?
Cloudflare grouped AI crawling by purpose: about 80% was for training models, 18% for search and 2% for actions triggered by a user. Training crawls take content in bulk, while search and user crawls fetch pages to answer a specific question.
Which AI company crawls the most per visit it sends?
In Cloudflare’s July 2025 data, Anthropic’s crawlers made about 38,066 requests for each visit referred, OpenAI’s about 1,091, Perplexity’s about 195, Microsoft’s about 41 and Google’s about 5. The ratio compares crawling to referrals, and it does not measure quality.
Does robots.txt stop AI crawlers?
It is a request, not a lock. Well-behaved crawlers follow it and others may not. Decide your policy by purpose: you may want to be found by search crawlers while declining training crawlers, and list each by its published user agent.
Does Google need a special file to show my site in AI Overviews?
No. Google says no special file or markup is needed, and pages must be indexed and eligible to show a snippet. The same applies to llms.txt, which Google does not use for AI Overviews.
How common is llms.txt?
Adoption is low. The datasets we cite put it at roughly 10% in one sample of sites, and about 2% of crawled sites in the Web Almanac, depending on what was sampled. SE Ranking found no link between having an llms.txt file and being cited.
SOURCES
Sources and notes
6 sources on this page: 4 independent or official (grade A) and 2 vendor or agency studies with a disclosed method (grade B). Numbers in brackets in the text match the numbers below. We read every figure on the source’s own page. How we grade sources.
- Platform dataGrade APublished 2025-08-29Read the source
- What we use:
- Crawl-to-referral ratios by platform and the purpose of AI crawling.
- Period:
- January to July 2025 (purposes: past 12 months)
- Sample:
- Cloudflare Radar crawl and referral data; referral data comes from news-related customers.
- Read with care:
- Purpose classification relies on operator disclosures; referral data may not represent the wider web.
- What we use:
- None of the major AI crawlers it studied render JavaScript.
- Period:
- 2024
- Sample:
- Crawler request logs across Vercel’s network.
- What we use:
- How common titles, descriptions, canonicals, structured data, robots.txt and llms.txt are across the web.
- Period:
- 2025 crawl
- Sample:
- A crawl of millions of public websites; figures are for desktop and mobile pages.
- Read with care:
- Last updated 2026-06-09.
- Vendor studyGrade BPublished 2025-11-07Read the source
- What we use:
- llms.txt adoption among a sample of domains, and the lack of a measurable citation effect.
- Period:
- 2025
- Sample:
- About 300,000 domains; Spearman correlation, XGBoost and SHAP analysis.
- Read with care:
- The authors say results are context-dependent and may be outdated within weeks.
- What we use:
- What Google says is and is not required to appear in AI features.
- Period:
- Documentation, last updated Dec 10, 2025
- Sample:
- Google’s own guidance for AI Overviews and AI Mode.
- What we use:
- AI Overview prevalence and AI referral share by industry; ChatGPT’s share of AI referrals; most-cited domains by industry.
- Period:
- May to Sept 2025 (AI Overviews: Sept 15 to Oct 12, 2025)
- Sample:
- 13,770 enterprise domains in 10 industries; 17M AI responses and 100M+ citations; AI referral data from 1,215 domains (3.3B sessions, 35.7M AI sessions); 21.9M US Google searches for AI Overviews.
- Read with care:
- US data. AI Mode traffic is excluded because Google Analytics groups it with organic Google traffic. Some figures appear twice with different rounding; we use the more precise one.
Preferred sources on Google
Make Webtec a preferred source on Google.
Pick us once, and Google shows you more of our research and guides in Top Stories when we publish. It takes one click and you can change it at any time.
A LIVING REFERENCE
Refreshed every quarter.
We re-check every source in January, April, July and October, add new studies and retire old ones. This is Edition 1 (October 15, 2026); the next is due January 2027. Changes are logged on the methodology page.
