Cloudflare Goes To War Against AI Using A One-Click Solution To Block Their Web Scrapers

Cloudflare AI

The internet was once a crowded ecosystem with way too many noises and chatters.

To look for information, people were used to use internet search engines, visit their favorite websites, use social media platforms, in an attempt to scout and locate. But with AI, things get disrupted, because the heavy lifting of looking for information has been streamlined.

AI chatbots provide a simple chat box where users can ask whatever they want, and let the AI look for the information on users' behalf.

It's easy, and it's that simple.

While many users on the internet are happy, and this can be seen with the immense popularity of popular Large Language Models AI, like ChatGPT, not everyone is particularly pleased.

And this is where Cloudflare steps in.

Previously, Microsoft AI executive said that everything on the open web is "freeware", which means that Microsoft is free to steal any and all content people have published on the web to power its products.

Google also once said that it has the "right" to collect publicly-available information for its AI to work.

This disappointed a lot of content creators, artists and others, who wish to keep their work their own, and not to be used for AI training.

While the AI companies said that people have the options to block their crawlers from ever scraping their content, things in practice don't really reflect their policy.

When all hope was kind of lost, Cloudflare introduces a way to prevent these AI from scraping web content, using a one-click solution to thwart them all.

Cloudflare, the global internet service and hosting company, powering roughly 20% of all web traffic, has been offering services that include DDoS protection from attacks and bot verification checks on websites.

In all, Cloudflare has been instrumental in improving the general quality of the World Wide Web, using its massive server infrastructure as a vast security layer for companies of all shapes and sizes.

And here, to combat AI and preserve a safer internet for creators alike, the company launched what it refers to as an "easy button" to block all AI bots.

Available to all Cloudflare users, including those on its free tier, Cloudflare wants users to declare "AIndependence."

Cloudflare AI

Cloudflare said on its blog, that the system allows users to opt-in to block AI bots and crawlers from accessing websites, effectively preventing AIs from OpenAI, Microsoft, Google, and others from stealing web content for free.

After surveying its users, Cloudflare shared data that over 80% of its users wanted the ability to shut AI companies from stealing their content.

"We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly." Cloudflare continued, "To help, we’ve added a brand new one-click to block all AI bots. It’s available for all customers, including those on the free tier."

Generative AI training content is becoming lucrative and valuable to companies like Google and Microsoft. Google reportedly paid upwards of $60 million dollars for access to all of reddit's content for training its models, which also hilarious resulted in sarcasm and trolling appearing in Google search results.

Cloudflare AI

Tech giants are rewriting the rules on web scraping, blaming unnamed third parties for disregarding robots.txt, and seemingly claiming the right to reuse anything posted anywhere for their AI products.

This creates issues, because this can prevent human content creators from effectively monetizing their works and earn a living.

Due to the fact that AI doesn't really understand the content it is creating, and that it's still riddled with hallucinating problems, there is a huge concerns over the quality of the content the tools produce.

Model collapse happens, and because of this, human intervention is needed.

And in this case, at least according to Cloudflare, a switch is needed.