5 Ways to Block AI Crawlers — What Actually Works (October 2026)

No affiliate links on this page — every link points to official docs or reporting, nothing sponsored.

Everyone's blocking AI crawlers this month. Robots.txt files are getting edited at 2am, Cloudflare dashboards toggled by people who've never opened the Security tab. Here's the uncomfortable bit: most of the blocks don't actually block anything. We compared the five real options — what each stops, what each costs, and the catch nobody mentions.

The short version

1. robots.txt — the polite "please don't"

A text file at your domain root saying which crawlers may fetch which paths. It stops the well-behaved ones: GPTBot, OAI-SearchBot, CCBot (Common Crawl), Google-Extended all check it and honour it. If your worry is "my content training the next model," one Disallow line does most of the work.

The catch: scrapers running headless browsers, SEO tools, and the "definitely a real Chrome browser" crowd don't read it. Blocking AI training and blocking scraping are two different problems — this only solves the first.

Best for: everyone. Two minutes, free, catches the biggest training pipelines. Just don't believe you're done.

2. Server-level user-agent blocks — free, brittle, and kind of fun

Nginx/apache rules (or a plugin doing the same) that 403 requests whose user-agent matches a known bot list. Stops whatever's on your list, today — GPTBot announces itself, most honest ones do.

The catch: a treadmill. Bots that want in rotate user agents; every few weeks a "new" AI scraper appears using a generic Chrome UA and your blocklist becomes decoration. Block the wrong string and you nuke your own uptime monitor or a reader's RSS app.

Best for: small static sites where you control the server config. A supplement, not a strategy.

3. Cloudflare Bot Fight Mode — the free bouncer

One toggle on Cloudflare's free plan serving JavaScript challenges (Managed Challenge) to traffic its heuristics flag as automated. It stops the rude ones — scrapers, credential-stuffing scripts, botnet-adjacent junk that doesn't identify itself. It doesn't read robots.txt or user agents; it watches behaviour. The first method on this list that actually blocks rather than asks.

The catch: blunt. It can't distinguish "AI training crawler I'd allow" from "scraper stealing my content" — binary: looks automated, gets challenged. Legit users on odd networks occasionally eat challenges too, and sophisticated scrapers with real browser automation sometimes pass. It can also interfere with legitimate API traffic behind the same zone — people discover this at the worst possible moment.

Best for: anyone on Cloudflare's free tier getting hammered by obvious junk bots. Turn it on, watch Security events for a week, judge the false-positive rate.

4. Super Bot Fight Mode — the $25 bouncer with a guest list

The Pro-plan ($25/month at time of writing) upgrade: definite-bot blocking instead of mere challenges, static-resource protection, and — the important part — a verified-bots allowlist so known-good crawlers bypass challenges.

The catch: $25/month is real money for a side project, and "definitely automated" is a confidence score, not a fact. If your goal is specifically AI training control, this still isn't a scalpel: it sees automation, not intent. Pair it with robots.txt; don't expect it to replace that file.

Best for: sites where bot bandwidth shows up on the bill or in the analytics.

5. WAF custom rules + behavioural blocking — the actual answer, at a price

Cloudflare WAF custom rules (Business, $250/month) or your own stack: rate limiting per IP/ASN, blocking datacenter ASNs, TLS/JA3 fingerprinting, tarpitting repeat offenders. Blocking by behaviour instead of by name.

The catch: the price — $250/month or your own engineering hours with fail2ban, CrowdSec, or CDN edge rules. Tuning is the whole game: too aggressive and you block a university NAT with 400 students; too loose and you're back to decoration. This is a part-time job, not a toggle.

Best for: businesses where scraping has a dollar cost — paywalled content, usage-billed APIs, anyone whose logs show the same datacenter IPs chewing through pages daily.

What we'd actually do

Personal blog / side project: robots.txt for the polite training bots, plus free Bot Fight Mode if junk traffic actually bothers you. Don't pay $25/month to protect a blog nobody's scraping.

Content site with real traffic: add Super Bot Fight Mode on Pro, read the Security events weekly for the first month, keep robots.txt maintained — the polite bots are still the biggest training pipelines and the cheapest to stop.

Where scraping costs money: behavioural WAF rules, and accept it's now a maintenance task like backups. There is no set-and-forget.

One more thing, worth repeating: before blocking anything, decide whether you want to block training or fetching. Blocking ChatGPT-User doesn't stop model training — it breaks "summarise this URL" for your own readers and nukes your ChatGPT referral traffic. Block the training bots. Leave the user-initiated fetches alone.

FAQ