Cloudflare Just Opened the Bot-Control Spigot. Your Ad Revenue Is Being Stolen by Training Bots.
Cloudflare launched granular AI Bot Management this week. The feature separates three categories of bots: Search (Google, Bing, DuckDuckGo), Agent (Claude Cowork, ChatGPT Work), and Training (bulk content scraping for model training). Starting September 15, 2026, new domains will automatically block Agent and Training bots on ad-supported pages.
If you run an ad-supported blog or content site, this is the fix you've been missing. AI training bots are scraping your content, burning your bandwidth, and stealing your ad impressions. Robots.txt can't stop them. It's a suggestion, not a law. Cloudflare's new defaults can.
The problem: AI training bots eat 20-30% of your ad revenue
You run a blog. It generates $5K/month from ads. You publish 100 posts. A human reads a post, the page loads, an ad impression fires, you get $0.50–$2.00 depending on your CPM and geography.
An AI training bot crawls the same post. It reads the full text, doesn't render the ads, doesn't click anything. It's literally just vacuuming your words into a training dataset. You get zero revenue. Cloudflare's data shows this represents 20-30% of traffic to ad-supported sites.
Scale: if you're pulling 100K monthly users and 25% are training bots, 75K human visitors are seeing ads. If your CPM is $10 and your traffic is 400K pageviews, you'd expect $4K revenue. But if 25% of those pageviews are bots, you're actually serving ads to 300K pageviews, so you get $3K. That's $1K/month you're losing to bots.
Multiply across a portfolio of 10 sites, and you're leaving $10K/month on the table.
Why robots.txt doesn't work
Robots.txt is a suggestion. It says "please don't crawl this." Most legitimate bots respect it (Google, Bing). AI training bots ignore it. They're designed to scrape everything, everywhere, because the entire internet is their training dataset.
Google respects robots.txt because Google's business model depends on trusting webmasters (they send SEO traffic). Training bots don't depend on anything: they're pulling data to train models that will eventually disintermediate you.
You can block specific user agents in robots.txt:
User-agent: GPTBot Disallow: /
But by the time you block ChatGPT's crawler, there are 50 other training bots you don't know about. And OpenAI can change the user agent tomorrow.
The root problem: you have no enforcement. You're asking nicely. They're ignoring you.
What Cloudflare Bot Management actually does
Cloudflare's bot management categorizes traffic into three buckets:
Search bots: Google, Bing, DuckDuckGo, other indexers. These bots send you traffic. They're valuable. Allow them.
Agent bots: Claude Cowork, ChatGPT Work, similar AI agents that humans are using to browse the web. These are legitimate: if someone is paying for ChatGPT Plus and using it to research, they want their agent to access your site. But they also don't generate ad impressions. Allow selectively.
Training bots: Bulk scrapers collecting data for model training. They generate no revenue and no traffic. Block them.
The new default for new domains (starting September 15): block Training and Agent bots on ad-supported pages. Allow Search.
Here's what you do this week
Step 1: Audit your current bot traffic
Log in to Cloudflare. Navigate to Analytics > Bot Traffic. Check your last 30 days. Sort by bot type. You'll see a number like "24% of traffic is Training bots" or "18% is Agent."
Screenshot it. This is your leverage for reprioritizing bot management.
Step 2: Set your bot policy
Go to Cloudflare Dashboard > Security > Bot Management. You'll see options for each bot category.
For ad-supported sites:
- Search: Allow (these send you traffic)
- Agent: Set to "Definitely automated" (you can block or challenge; I'd challenge, let legitimate users through)
- Training: Block
For sites where you want your content accessed by AI (e.g., documentation sites you want Claude to read):
- Search: Allow
- Agent: Allow
- Training: Block (you get no value from uncompensated training data scraping)
Step 3: Document your decision
Add a note to your robots.txt or site policy explaining which bots you allow and why:
# AI Bot Management (Cloudflare) # Allowed: Search (Google, Bing, etc.) # Blocked: Training bots (model training scrapers) # Challenged: Agent bots (ChatGPT Work, Claude Cowork)
Step 4: Act before September 15
If you're on an existing domain (not new), the September 15 defaults don't apply to you. You need to set your policy manually. Do it now, before you forget.
The honest complication
Some AI services are legitimate. If you're a software documentation site and Claude Cowork users benefit from your docs being accessible, you might want to allow Agent bots. If you're a research publication and you want your work in training datasets, allow Training bots (and negotiate compensation if you can).
But the default assumption should be: "I wrote this for humans to read and interact with, not for AI companies to vacuum up for free."
Cloudflare's bot management lets you make that choice. Use it.
The September 15 deadline
Mark your calendar. On September 15, Cloudflare's default changes for new domains. If you're on an old domain, your settings don't change. But that's an excuse to audit and update your policy.
Existing domains with bot management already enabled won't change. Cloudflare respects your existing settings. New domains get the secure default.
If you're undecided, wait until September 15 and see what happens with the new default. Then decide if you want to adjust for your use case.
Author
Lukas
@lukcombinator