Cloudflare’s Pay Per Crawl: A New Era for Content Monetization

AI companies have been crawling the internet at scale, gathering information without paying the people who created it. For publishers, this has resulted in less traffic and fewer rewards for their hard work. Cloudflare’s new Pay Per Crawl program aims to change that. With over 24 million using the service, this shift marks a new phase in digital content monetization — how digital content is valued and paid for. 

Why AI Crawling Has Become a Problem for Publishers

AI models like those from OpenAI rely on large amounts of online content to train their systems. That means they crawl millions of websites, typically without asking or giving credit. TollBit reports AI engines send 96% less referral traffic compared to Google and that scraping outpaces user visits.

Publishers have raised concerns about this setup. They put time and money into creating content. However, AI tools use it freely and send little traffic back. This makes it more difficult for websites to earn money or grow their audience. Without clear rules or agreements, many content creators are losing control over how their work is used. 

What is Pay Per Crawl?

Pay Per Crawl is Cloudflare’s new feature that gives publishers more control over who can access their content and how they do it. Cloudflare powers nearly one-fifth of all websites whose reverse proxy service we know, representing 19.4% of all websites. This update, then, could reshape how AI bots gather data across a large part of the internet. 

  • Digital monetization is how online platforms turn traffic or content into revenue. Examples include subscription services like Netflix, ad-supported models like YouTube, or pay-per-use systems such as Cloudflare’s new Pay Per Crawl.

Default Blocking for AI Crawlers

Cloudflare now blocks AI crawlers by default for all new users, meaning tools like OpenAI’s GPTBot can’t access your site unless you let them. For existing users, this feature can be turned on with just one click in the settings. 

This default setting shifts the balance. Instead of assuming AI companies have open access to web content, they’ll need permission. It gives publishers a clearer path to decide what’s shared and what isn’t, establishing a new standard for web control and privacy. 

Monetization Options for Publishers

With Pay Per Crawl, publishers can choose from three content monetization strategies: allow AI crawlers for free, charge a fee, or block them altogether. This flexibility lets websites decide what’s worth sharing and what needs protection. 

Smaller sites may block crawlers to keep their content exclusive. Larger publishers might see value in charging access fees. Over time, this model leads to more structured relationships between publishers and AI platforms. 

Content scraping has already been linked to an estimated 2% loss in yearly revenue for some sites. In contrast, high-traffic websites can earn between $50,000 and $200,000 a month from AI crawlers under this new model. 

  • Related Article: How to Track AI Traffic in GA4: Explorations & Primary Channel Group

    AI tools like ChatGPT and Perplexity are changing how users find content—and how it shows up in analytics. Learn step-by-step methods to track AI-driven traffic in GA4, spot trends, and adjust your strategy for this new reality.

How the 402 “Payment Required” Status Works

Cloudflare is using the HTTP 402 status code, “Payment Required,” to ensure Pay Per Crawl works. This status code was originally created for digital cash or micropayment systems, but it has seldom been used until now. 

When an AI crawler hits a site with Pay Per Crawl enabled, it either has to pay or gets a pricing notice in return. This puts a price tag on access instead of leaving things open-ended. 

AI bots drove an 86% increase in general invalid traffic during the second half of 2024. This system offers a new way to manage and monetize that kind of activity. 

  • The process depends on the platform. Cloudflare’s Pay Per Crawl is currently in a private beta. You can apply to join the program on their Get Early Access page.

The Bigger Picture: AI, SEO, and the Future of Web Content

AI-generated answers and search tools are changing how people find and use information online. In fact, monthly traffic to Generative AI services grew by 251% over the past year (February 2024 to March 2025). As traffic patterns shift, publishers are beginning to ask bigger questions. They want to know about the role of SEO, web visibility, and long-term content value in an AI-first world. 

  • Related Article: The Future of SEO and Web Architecture in the Age of AI Overviews

    AI overviews are changing how search engines understand and display content, forcing brands to rethink site structure. See why intent-driven pages, schema, and a clear hierarchy are becoming essential for ranking, and for staying visible in the age of AI search.

Declining Referral Traffic and Publisher Impact

AI search tools generally give users direct answers without linking back to the original source. This has led to a drop in traffic for many publishers. OpenAI’s GPTBot, for example, made 569 million requests in a single month through Vercel, while Claude sent 370 million. Yet few of those requests translated into clicks. 

Bots now make up 80% of all web visits. Of those, about 13% are from AI bots like GPT, while only 8% come from traditional sources such as Googlebot. As a result, some publishers have seen traffic drop by as much as 90%. Across the board, traffic to the world’s 500 most-visited publishers has dropped 27% year-over-year. That's an average of 64 million fewer visits every month. 

  • Related Article: How AI Chooses What to Cite and What That Means for Your Content Strategy

    AI tools are favoring trusted, human-driven sources like Reddit, Wikipedia, and LinkedIn. See how citation patterns are shifting and what steps your brand can take to earn more AI mentions.

Ethical Implications for AI Training Models

There’s a growing debate around whether AI companies should be able to use public content without paying for it. Many publishers argue that if AI tools depend on their work, there needs to be some form of licensing or compensation.

Legal action has already begun. High-profile cases like The New York Times vs. OpenAI are shaping how courts view AI training practices. Meanwhile, AI search engines send 96% fewer referrals than traditional search. The gap between content use and value return is getting tougher to ignore. 

Shifting Power Toward Center Creators

Pay Per Crawl gives content creators more say in how their work is used. Instead of being mined for free, publishers can now decide where to share, block, or charge. 

This shift could rebalance the current system. If AI companies want dependable, high-quality data, they might need to start forming formal partnerships with the people who create it. 

Should Publishers Embrace Pay Per Crawl? Pros and Cons

Pay Per Crawl gives publishers more control, but it also comes with trade-offs. Charging AI bots may bring in new revenue, but it could also limit SEO visibility and discovery, particularly for smaller sites trying to grow their audience. 

For large media outlets, monetization could be a wise move. For niche blogs, blocking or charging may reduce exposure. Publishers lost an estimated $2 billion in ad revenue because of AI-powered search features in 2024, but that same year, bot-driven traffic cost businesses $238.7 billion. Deciding when and how to engage is an important call. 

  • The biggest mistakes include misunderstanding your audience, relying on a single revenue stream, and stuffing pages with too many ads. Others include testing too many ad formats at once and pushing paywalls or subscriptions before building trust, which can turn users away instead of generating revenue.

What This Means for the Future of Content Monetization

AI-driven tools are changing how content is discovered, read, and rewarded. Cloudflare’s Pay Per Crawl is an early sign that publishers might finally have more to say and also more ways to earn. As these changes take shape, the leading question is how to respond. 

“Pay-per-crawl content could help transform SEO from a game of ranking to one of both access & value attribution. Publishers & websites alike can now structure their sites for monetization by thinking about their content as data with value as opposed to just content that will be crawled. Think about your content that provides the most value and make sure you are getting the right attribution and monetization.”
— Lawson Robinson, Eyeful Media, Sr Manager, SEO

Key Takeaways:

  • AI bots now account for a large portion of web traffic but offer little value in return.

  • Cloudflare’s Pay Per Crawl gives publishers control over access and monetization.

  • The HTTP 402 status code makes it easier to ask AI crawlers to pay for content. 

  • Referral traffic is down, especially for niche publishers.

  • Tools like Pay Per Crawl might push AI firms toward paid licensing deals.

Action Items:

  • Review your site’s AI bot settings through Cloudflare or your hosting platform.

  • Decide which pages, if any, should be monetized or blocked from AI crawlers.

  • Monitor how much of your traffic is coming from bots vs. human visitors.

  • Explore options for charging or partnering with AI platforms.

  • Reevaluate SEO strategies in light of declining organic referrals. 

Partner with Eyeful’s SEO experts to lock in your content monetization strategy. 

Content, AI Search, SEOTodd O'Rourkeseo, ai search, content marketing