Free, instant, no sign-up

AI crawler checker

See which AI crawlers your robots.txt lets in: ChatGPT search, Claude, Perplexity, Google's Gemini control and the training bots, with what each one does and what blocking it means.

We read only your robots.txt file. No sign-up, no email.

What the checker looks at

It reads one file, your website's /robots.txt, and applies its rules to 23 crawlers and control tokens from OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple, Meta, Amazon, Mistral, DuckDuckGo and Common Crawl. Each crawler's name and purpose comes from its vendor's own documentation.

  • The rules are read the way crawlers read them. We follow the robots.txt standard (RFC 9309): a crawler obeys the group that names it, otherwise the * group, and the most specific matching rule wins.
  • Allowed, partly blocked or blocked. We test your home page and a few typical content paths. Rules that only cover admin, cart or account folders still count as allowed.
  • A missing or broken file is reported too. No robots.txt means crawlers may read everything; a robots.txt that returns a server error means crawlers that follow the standard stop crawling until it works again.

Search, training and on-request fetches: what to allow

  1. Search crawlers can send you visitors. OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot and PerplexityBot put pages in AI answers with links. If you want to appear there, allow them.
  2. Training crawlers are a business choice. GPTBot, ClaudeBot, CCBot and others collect pages that may be used to train models. OpenAI and Anthropic use separate crawlers for their search features, so blocking their training crawlers does not block their search crawlers.
  3. Google-Extended does not affect Google Search. Google says the token "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Blocking Googlebot, on the other hand, removes you from Google Search.
  4. On-request fetches may ignore robots.txt. OpenAI, Perplexity, Google, Meta and Amazon say fetches made because a person asked about a page may not follow it.

What robots.txt cannot tell you

It is a request to well-behaved crawlers, not a lock.

  • Firewall and CDN rules (for example a "block AI bots" setting) can turn crawlers away before they read robots.txt. The checker cannot see those settings.
  • Being allowed is not the same as being cited. AI answers depend on your content and on the other sites that describe you.
  • robots.txt does not protect private content. Anything that must stay private needs a login.

How to change what the checker finds

Edit the robots.txt file at the root of your website, then run the check again.

  • To allow a crawler that a * rule blocks, add its own group, for example User-agent: OAI-SearchBot followed by Allow: /.
  • To block a training crawler only, add User-agent: GPTBot followed by Disallow: /, and leave the search crawlers alone.
  • On WordPress, Shopify and other platforms, robots.txt is often generated: use the platform's setting or an SEO plugin rather than uploading a file.
  • If your robots.txt returns a web page or an error, check your firewall or CDN settings first.

Frequently asked questions

Is the AI crawler checker free?

Yes. There is no sign-up and no email. It reads your public robots.txt file and shows the result on the page.

Should I block GPTBot?

It depends on whether you are happy for your public pages to be used to train OpenAI's models. Blocking GPTBot does not block OAI-SearchBot, which OpenAI uses for ChatGPT search, so you can block one and allow the other.

Does blocking Google-Extended remove me from Google Search?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search; it controls whether Google may use your pages for its Gemini models. Blocking Googlebot is what removes a site from Google Search.

Why does the checker say a crawler is partly blocked?

Your rules stop it from reaching some of the typical pages we test, such as the home page, a blog post or a product page, but not all of them. The report lists the paths your file disallows for that crawler.

Do I need an llms.txt file as well?

Not for Google: Google says llms.txt isn't needed for Google Search. Other assistants may read it, but it does not replace robots.txt or good page content.

Can you fix this for me?

Yes. Our AI search optimisation service sets up robots.txt, structured data and the pages and listings that AI assistants read, and checks the result.

Want AI assistants to find and recommend you?

Letting the right crawlers in is step one. We fix the pages, structured data and business listings that AI assistants read, then check what they say about you.

Book a free 30-minute project callAbout AI search optimisation