The LLM war rages on fiercely into 2026, with tech companies battling for supremacy in intelligence, speed, multimodal capabilities, and real-world utility.
While OpenAI started the trend when it released ChatGPT, which sparked the development of many other commercial large language models (LLMs), current benchmarks show no single undisputed leader anymore. This is because at the heart of what makes modern chatbots increasingly smart and useful to users is their ability to browse the web in real-time.
Static training data alone limits LLMs to outdated knowledge, but integrating live internet access, through features like Retrieval-Augmented Generation (RAG), agentic browsing, and dedicated search tools, allows them to fetch current information, verify facts, and deliver accurate, up-to-date answers.
This shift transforms chatbots from clever pattern-matchers into dynamic research assistants capable of handling news, market trends, weather, research, or complex queries that require fresh data.
With LLM-powered chatbots having more users and engagements, the result is that, the trend affects the overall health of the web.
With people looking for answers through LLMs, websites are getting less and less traffic.
And Anthropic is mostly to blame, according to a 'Cloudflare Radar: 2025 Year in Review' report from Cloudflare.
In the sixth annual review of the Internet trends and patterns Cloudflare observed throughout 2025, based on its expansive network view, a significant shift in how AI platforms interact with the web, particularly through the lens of the crawl-to-refer ratio.
This metric measures the number of times a platform's bots request HTML pages (crawling) compared to the number of times it sends referral traffic back to those sites via user clicks or visits.
A high ratio indicates heavy data consumption with minimal return traffic, disrupting the traditional "grand bargain" of the web where crawlers index content in exchange for driving valuable human visitors to publishers
.
Among leading AI and search platforms, Anthropic stands out with the highest crawl-to-refer ratio observed throughout 2025.
"Anthropic had the highest crawl-to-refer ratio among the leading AI and search platforms," Cloudflare said.
Early in the year, this ratio reached extreme levels as high as 500,000:1, meaning ClaudeBot crawled hundreds of thousands of pages for every single referral sent back. While volatile in the first half due to sparse referral traffic, it later stabilized between approximately 25,000:1 and 100,000:1 after May.
This pattern reflects Anthropic's ClaudeBot activity, which roughly doubled in the first half of 2025 before declining gradually, ending the year about 10% higher than its starting level.
In comparison, other platforms show far more balanced or lower ratios.
OpenAI's ratio was spikier, peaking around 3,700:1 in March, with overall figures in the hundreds to low thousands. Perplexity maintained the lowest among major AI players, generally below 400:1 and often under 200:1 later in the year, despite its PerplexityBot crawling growing 3.5 times over the year. Traditional search engines like Google saw ratios rising to 30:1 before falling back toward 3:1, while Microsoft Bing stayed in a cyclical 50:1 to 70:1 range.
These differences underscore how conventional search prioritizes sending users to sources, whereas many AI systems focus more on data extraction for training or on-platform answers.
Recent data extending into early 2026, as reported by Business Insider, indicates the imbalance persists or even worsens for some players.
Anthropic's crawling increased further in early January compared to September 2025, with ratios deteriorating despite earlier promises of growing referral traffic from features like web search in Claude. OpenAI shows similar trends of more aggressive crawling without proportional referrals back to the web. This has sparked concerns among publishers, who face higher server costs from bot traffic, sometimes doubling cloud bills, while receiving little in return as AI tools increasingly provide direct answers instead of links.
The broader implications are clear: AI crawlers now account for a notable portion of automated web traffic, with training bots dominating over search or user-action purposes.
While "user action" crawling (bots visiting sites in response to user queries) surged over 15 times in 2025, overall the ecosystem reveals a growing tension.
Content creators are left questioning the sustainability of freely providing data to AI companies that prioritize model improvement over ecosystem support.
As 2026 unfolds, this crawl-heavy behavior from Anthropic and others continues to fuel debates about fair compensation, robots.txt enforcement, and the future balance between innovation and the open web's health.