AI Crawlers: Why Blocking All of Them Costs You Traffic

The usual advice about AI crawlers is a block list. Paste 12 user agents into robots.txt, disallow everything, done. It takes 2 minutes and it quietly costs some sites the one channel that was still growing.

Because AI crawlers are not all doing the same job. Some are collecting text to train a future model. Some are building the index that decides which sites get cited in AI answers today. And some are fetching a page because a human just asked a question about it.

Block all 3 with one rule and you have opted out of being mentioned in order to avoid being trained on.

Rather skip the DIY? The AI search optimization service covers SEO, AEO, and GEO in one engagement, starting with a citation audit.

AI Crawler Categories

Crawler identity, stated purpose, robots.txt control, and whether a page can appear in an AI answer are separate fields. A single allow-or-block label loses the part of the policy that actually matters.

A robots.txt rule is a policy signal, not an authentication system. Verify status codes, redirect chains, canonical tags, rendered content, and bot-mitigation behavior separately before concluding that a crawler can or cannot use a page.

Reading a user agent before you block it

Naming conventions are inconsistent between operators, which is most of why these get lumped together. A few patterns help.

• Names ending in -User generally indicate a user-triggered fetch. Somebody asked a question and the assistant went to look.
• Names containing Search generally indicate index building for answers and citations.
• Plain operator names tend to be the training crawlers, though not universally.
• Extended variants from large search operators usually exist specifically as a training opt-out that leaves normal search crawling untouched.

That last one is worth care. Blocking a search engine’s normal crawler and blocking its AI training variant are completely different actions, and confusing them can remove you from ordinary search results. Check which name you are disallowing.

Operators publish their current user agents and IP ranges in their own documentation, and the lists change often enough that a block list copied from a blog post is usually out of date. Take the names from the source, not from an article, including this one.

Writing the rules

A policy reflecting the categories above looks roughly like this, with the actual agent names filled in from each operator’s current documentation:

User-agent: SomeTrainingBot
Disallow: /

User-agent: SomeSearchBot
Allow: /

User-agent: SomeAssistantUserAgent
Allow: /

Order matters less than specificity: most crawlers match the most specific applicable block, so a broad wildcard rule below a named one usually does not override it.

Test the result rather than assuming it. Fetching your own robots.txt and reading it as the crawler would is a 30-second check that catches the common error of a stray wildcard disallowing far more than intended.

robots.txt asks AI crawlers nicely and cannot make them listen

The critical limitation, and the reason a robots.txt-only strategy disappoints people who expected enforcement.

robots.txt is voluntary. It is a published preference that well-behaved AI crawlers honor and badly-behaved ones ignore, and there is no technical mechanism inside the standard that compels anything. Major operators generally comply, because non-compliance is a reputational problem for them. Scrapers with no reputation to protect do not.

If the goal is actually to stop a fetch rather than to state a preference, enforcement happens at a different layer:

• Your CDN or WAF can block by user agent, IP range, or bot signature, and it refuses the request rather than asking nicely.
• Rate limiting handles the aggressive crawlers whose real cost is server load rather than the copying.
• Authentication is the only reliable boundary. Content behind a login is not available to any crawler, compliant or otherwise.

Verification matters here too. User agent strings are trivially forged, so a request claiming to be a well-known AI crawler may be anything at all. Operators publish IP ranges precisely so you can confirm, and blocking on an unverified user agent string alone catches the honest bots while missing the dishonest ones.

The decision worth actually making

Strip away the technical detail and there are 3 coherent positions. Most sites should pick 1 deliberately rather than arriving somewhere by copying a snippet.

Allow everything. You want maximum reach, you accept your text contributes to model training, and you would rather be cited than protected. Reasonable for most publishers whose business depends on being found.

Block training, allow retrieval. You object to being training material but still want to appear in AI answers. This is the position most publishers actually hold when asked, and it takes the most care to configure because it requires distinguishing the categories correctly.

Block everything. You have a specific reason: the content is proprietary, licensed, or paid for, and appearing in an AI answer cannibalizes the reason someone would visit. Legitimate, and worth adopting on purpose rather than by accident.

The one position that is never deliberate is blocking retrieval while allowing training, which is the exact outcome of pasting an unexamined block list. It gives away the thing you might have objected to and forfeits the thing you wanted.

What this does not tell you

Honest boundaries, because this area is full of confident claims that outrun the evidence.

Nobody outside these companies knows how much AI referral traffic a given site is losing or gaining from these choices. The measurement is genuinely hard: assistants often do not pass a referrer, so the traffic that does arrive is frequently indistinguishable from direct.

Blocking a training crawler today does not remove your content from models already trained. It affects future collection, and nothing published before the block goes back in the box.

The user agent list changes without notice as operators add, rename and split crawlers. Any configuration here needs revisiting a few times a year, and a list that has not been reviewed since last year is probably already wrong.

And none of this is a legal position. Whether training on published content is permitted is being argued in courts in several jurisdictions, and robots.txt is a technical preference rather than a licence. If the answer matters commercially, that is a question for a lawyer and not a configuration file.

Questions people ask next

Decide what the fetch is for

The block-everything reflex treats all AI crawlers as extraction. It is a reasonable instinct and it flattens a distinction that costs real traffic, because the crawler building the index that cites you is not the crawler collecting text to train on, and the assistant fetching your page because someone asked about it is closer to a reader than to a scraper.

Sort them by purpose, decide your position once, and configure to that. If appearing in AI answers is the goal rather than avoiding them, the work sits on the optimization side rather than the blocking side, and the 2 are easy to accidentally work against each other.

Open your robots.txt. Check what you are already refusing.