Search Engine Optimization

AI crawlers and robots.txt: who to allow, who to block, and what it costs you

12 min readBy Jaskirat Singh Talwar

As of October 2026 every major AI company runs at least two bots: one that collects pages to train models and one that fetches pages to answer searches, plus a third kind that fetches a page because a person asked for it. robots.txt can stop the first two and generally cannot stop the third. For a business that wants enquiries the decision is simple: allow every search crawler, treat training crawlers as a policy choice that does not affect your visibility, and check your CDN logs monthly to confirm the bots you allowed are fetching the pages you want cited.

Key takeaways

  • Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent) does not remove you from that vendor's search or answer features. Google: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". OpenAI: "Each setting is independent of the others."
  • Blocking a search crawler does. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers"; Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results".
  • User-triggered fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Google's fetchers) "generally ignore robots.txt" because a person requested the page. RFC 9309 calls robots.txt "not a form of access authorization".
  • Crawling is not free. Cloudflare's July 2025 data puts the crawl-to-referral ratio at 5.4 pages per referred visit for Google, about 1,091 for OpenAI and about 38,066 for Anthropic. Your CDN bills per request.

Three kinds of bot, three decisions

Many sites blocked "AI bots" wholesale in 2023 and 2024 and quietly removed themselves from the answers that now shape buying research. The vendors have since split their bots:

  1. Training crawlers collect pages for model training. Blocking them is a legal and commercial choice that changes nothing about how you appear in search or AI answers.
  2. Search crawlers build the index behind ChatGPT search, Perplexity, Claude's search and Google's AI features. Blocking them removes you from those answers.
  3. User-triggered fetchers retrieve a page when a person asks about it or pastes a link. The vendors say these generally do not honour robots.txt.

RFC 9309, the 2022 standard, says robots.txt rules are "not a form of access authorization" and crawlers are "requested to honor" them; a rule stops a bot that chooses to obey, and anything else is a job for your CDN or firewall. Where no rule matches, "the URI is allowed", so a site with no AI rules already allows everything. This article is a companion to our GEO guide and the AI search visibility service; it covers only the crawler decision.

The crawlers, from the vendors' own documentation

Every cell comes from the vendor pages in the sources, checked in October 2026. Tokens are the User-agent values for robots.txt.

Vendor Token Purpose (vendor's words) robots.txt Blocking it removes
Google Googlebot Search index; "no additional requirements" for AI Overviews and AI Mode Honoured Google Search and every AI feature on it
Google Google-Extended Controls training of "future generations of Gemini models" and grounding; "doesn't have a separate HTTP request user agent string" Control token only Nothing in Search: "does not impact a site's inclusion in Google Search nor is it used as a ranking signal"
OpenAI GPTBot "used to crawl content that may be used in training our generative AI foundation models" Honoured; IP list published Training only
OpenAI OAI-SearchBot "used to surface websites in search results in ChatGPT's search features" Honoured; about 24 h to apply; IP list published ChatGPT search answers (site may still appear "as navigational links")
OpenAI ChatGPT-User User-initiated visits; "not used for crawling the web in an automatic fashion" "robots.txt rules may not apply" Little; not used for search eligibility
Anthropic ClaudeBot Collects web content that "may contribute to training" Honoured; Crawl-delay supported; IP list published Training only
Anthropic Claude-SearchBot "navigates the web to improve search result quality for users" Honoured Claude search visibility
Anthropic Claude-User Fetches pages when a user asks Claude a question Honoured, per Anthropic's general statement Visibility "for user-directed web search"
Perplexity PerplexityBot "designed to surface and link websites in search results on Perplexity"; not used for training Honoured; IP list published Perplexity search
Perplexity Perplexity-User Visits a page "to help provide an accurate answer" "generally ignores robots.txt rules" Little
Microsoft Bingbot Bing index; Copilot use is set per page with nocache and noarchive (below) Honoured Bing and Copilot
Apple Applebot, Applebot-Extended Siri, Spotlight and Safari search; data "may also be used to help train Apple foundation models"; -Extended opts out of training and "does not crawl webpages" Honoured; follows Googlebot rules if Applebot is not named -Extended: training only; pages "can still be included in search results"
Meta Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher "training foundation AI models" and indexing; Meta AI search; user-requested fetches First two honoured; the fetcher "may bypass robots.txt rules" Training; Meta AI search; little
Amazon Amazonbot, Amzn-SearchBot, Amzn-User Data "may be used to train Amazon AI models"; search, "does not crawl content for generative AI model training"; user actions Honoured; Amzn-User "may not follow all robots.txt directives" Training; Amazon search; little
Common Crawl CCBot Open web archive provided free "for the purpose of research and analysis" Honoured; Crawl-delay obeyed Presence in the Common Crawl corpus

Two notes. ByteDance's Bytespider was the largest AI crawler on Cloudflare's network in May 2024 and publishes no documentation we could find; treat it as training. Microsoft controls Copilot per page: Bing's September 2023 announcement says nocache content "may be included in Bing Chat answers" using only "URLs, Titles and Snippets", noarchive content "will not be included in Bing Chat answers, not be linked to in the answers" and is not used for training, and pages with either tag "will still appear in our search results". Bing Chat is now Copilot.

What blocking actually removes

The pattern is consistent: the training token and the search token are separate, and blocking one does not touch the other. Google's AI features page says there are "no additional requirements to appear in AI Overviews or AI Mode", so the only crawler decision that affects them is the one you already made for Googlebot; the preview controls (nosnippet, data-nosnippet, max-snippet, noindex) limit what Google shows in Search and AI features alike, after a recrawl. The one-way doors are the search crawlers: block OAI-SearchBot and ChatGPT stops citing you in search answers, block PerplexityBot and you leave Perplexity, block Claude-SearchBot and Anthropic says your visibility in Claude's search "may" fall.

The fetchers robots.txt cannot stop

Assistants now browse on request. Google's list of user-triggered fetchers (Gemini Notebook, Google-Agent, Google Read Aloud and more) says they "generally ignore robots.txt rules" because a user requested the fetch. OpenAI says the same of ChatGPT-User, Perplexity of Perplexity-User, Meta of Meta-ExternalFetcher, Amazon of Amzn-User.

Stopping these is bot-management territory. Cloudflare announced on 1 July 2025 that it was "changing the default to block AI crawlers unless they pay creators for their content", and in September 2025 added a Content Signals Policy that lets robots.txt state preferences for search, ai-input and ai-train separately, while noting the signals "are not technical countermeasures against scraping". For most service businesses, user-triggered fetches are visits you want: someone asked an assistant about you.

What crawling costs you

Cloudflare's network data is the only public scale we know of. In July 2025 it measured pages crawled per visit referred: Google 5.4, Microsoft 40.7, Perplexity 194.8, OpenAI 1,091.4, Anthropic 38,065.7. By purpose, 79% of AI crawler requests were for training, 17% for search and 3.2% for user actions. Between May 2024 and May 2025 GPTBot's request volume grew 305%, to 30% of AI-only crawler traffic, and of 3,816 top-10,000 domains with a robots.txt, 546 (about 14%) had rules aimed at AI bots.

CloudFront bills by request and data transferred; its free plan covers 1 million requests and 100 GB a month as of October 2026, which a few hundred HTML pages re-fetched by a dozen bots will not exhaust. Image-heavy sites and large catalogues are different: a training crawler fetching ten thousand URLs a month to send you nothing is a real line item, and Cloudflare's ratios are the argument for blocking training bots. Search bots are cheap by comparison and are the ones that send people.

Check your logs on CloudFront

Allowing a bot is not the same as being crawled; the proof is in the logs.

  1. Enable standard logging (v2) to Amazon S3. AWS states that "CloudFront doesn't charge for enabling standard logs" and there are "no additional charges for log delivery to Amazon S3", only storage. Logs usually arrive within an hour, can lag up to 24 hours, and become reliable about four hours after you switch logging on.
  2. Create an Athena table with AWS's published CREATE EXTERNAL TABLE statement for CloudFront standard logs. The user agent lands in cs_user_agent, URL-encoded.
  3. Run one query a month that classifies requests by bot and counts distinct pages:
WITH hits AS (
  SELECT "date", cs_uri_stem, sc_status,
    CASE
      WHEN cs_user_agent LIKE '%OAI-SearchBot%'    THEN 'OAI-SearchBot'
      WHEN cs_user_agent LIKE '%GPTBot%'           THEN 'GPTBot'
      WHEN cs_user_agent LIKE '%ChatGPT-User%'     THEN 'ChatGPT-User'
      WHEN cs_user_agent LIKE '%PerplexityBot%'    THEN 'PerplexityBot'
      WHEN cs_user_agent LIKE '%Perplexity-User%'  THEN 'Perplexity-User'
      WHEN cs_user_agent LIKE '%Claude-SearchBot%' THEN 'Claude-SearchBot'
      WHEN cs_user_agent LIKE '%ClaudeBot%'        THEN 'ClaudeBot'
      WHEN cs_user_agent LIKE '%Claude-User%'      THEN 'Claude-User'
      WHEN cs_user_agent LIKE '%Googlebot%'        THEN 'Googlebot'
      WHEN cs_user_agent LIKE '%bingbot%'          THEN 'Bingbot'
    END AS bot
  FROM cloudfront_standard_logs
  WHERE "date" >= DATE '2026-10-01'
)
SELECT bot,
       COUNT(*)                                         AS requests,
       COUNT(DISTINCT cs_uri_stem)                      AS pages,
       SUM(CASE WHEN sc_status = 200 THEN 1 ELSE 0 END) AS ok_responses
FROM hits
WHERE bot IS NOT NULL
GROUP BY bot
ORDER BY requests DESC;
  1. Verify the bots are genuine. User-agent strings can be forged; OpenAI, Perplexity and Anthropic publish crawler IP ranges as JSON. Anthropic warns that blocking by IP "may not work correctly or persistently" because it stops the bot reading your robots.txt.
  2. Compare against the pages you care about. A page no search crawler has fetched in 30 days will not be cited; check robots.txt, CDN or WAF rules, internal links and the sitemap, in that order. For the visits those crawls produce, see measuring AI search.

A recommended policy for a business that wants to be found

Write the decision into the file. This sample allows every search crawler explicitly and groups the training crawlers so the choice is visible; RFC 9309 allows several User-agent lines per group.

User-agent: *
Allow: /

# Search and answer crawlers: always allowed
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /

# Training crawlers: a policy decision, not an SEO one.
# Change to "Disallow: /" to opt out of training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Three additions. Check the layer in front of the file: Google asks that crawling be allowed "in robots.txt, and by any CDN or hosting infrastructure", and a 2023 "block AI bots" firewall rule overrides a 2026 robots.txt. Use page-level controls (Google's preview controls; Bing's nocache and noarchive) for pages you do not want quoted rather than blocking a crawler site-wide. And Crawl-delay is honoured by Anthropic and Common Crawl but not by Google, Apple or Amazon, so rate-limit heavy crawlers at the CDN.

What we did on impingesolutions.com

As of October 2026 our robots.txt is deliberately "allow all": User-agent: *, Allow: /, and the sitemap, with no separate rules for training crawlers. That is a decision, not an omission: the site exists to be found and cited, and a training opt-out would change nothing about that. The log check above is the method we recommend; we are not publishing bot counts for our own site here.

FAQ

If I block GPTBot, will my site disappear from ChatGPT?

No. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as independent settings: "Each setting is independent of the others." A site can disallow GPTBot and still be shown in ChatGPT search answers while OAI-SearchBot is allowed. Blocking OAI-SearchBot is what removes you.

Does blocking Google-Extended hurt my Google rankings?

No. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". It controls whether your content may train future Gemini models and be used for grounding. AI Overviews and AI Mode follow your Googlebot settings and preview controls.

Can robots.txt stop an AI assistant reading my page when a user pastes the link?

Generally not. Google, OpenAI, Perplexity, Meta and Amazon all document user-triggered fetchers that "generally ignore" or "may not follow" robots.txt because a person requested the page, and RFC 9309 calls robots.txt "not a form of access authorization". Stopping those fetches needs CDN or firewall rules, and for most businesses they are visits worth having.

Should I block AI crawlers to save bandwidth?

Block training crawlers if the cost or the principle matters to you: Cloudflare measured OpenAI crawling about 1,091 pages per referred visit and Anthropic about 38,066 in July 2025. Keep the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot); they are a small share of requests and the only ones that send people.

How do I know which AI bots visit my site?

Turn on access logs at your CDN and count requests by user agent each month. On CloudFront, standard logging to S3 is free apart from storage, and the Athena query above groups requests by bot and counts distinct pages. Confirm suspicious entries against the published IP lists.

Impinge Solutions runs the crawler check and the monthly citation panel as part of its AI search visibility service. To have your robots.txt, CDN rules and logs reviewed, request an estimate.

Sources

Are you ready to partner with us?

Our good-natured team is waiting to hear your business requirements.

Book a free project call

Stay informed with our newsletter