Free AI crawler checker
Is your website blocking AI assistants?
Type your web address. We read your robots.txt, the file that tells crawlers what they may read, and show which of the crawlers behind ChatGPT, Claude, Perplexity and Google it asks to stay out, with what to change.
What it checks
- Your robots.txt, read with the matching rules of the robots.txt standard (RFC 9309), for the crawlers behind ChatGPT, Claude, Perplexity and Google, plus Common Crawl
- Whether a server error, or a firewall turning our request away, is stopping crawlers reading the file
- Rules that came from Cloudflare's managed setting, and whether your website is behind Cloudflare
Two kinds of AI crawler
One kind fetches pages to answer a question someone is asking now, or to build the search index those answers draw on. Keep these out and an assistant may leave your website out of its answers. The other kind collects pages that may be used to train future models. Keeping those out is a choice about training, not about answers.
In our study of 5,000 Australian business websites, 4.3% of the 3,517 whose rules we could read asked at least one AI training crawler to stay out. 1.7% asked an answer crawler to stay out, and 53 of those 60 asked every crawler to stay out. 81 of the 99 sites that singled out AI crawlers used the block one Cloudflare setting adds, not a rule the business wrote.
Read the studyThe crawlers we check
What each one does, in its company's own words, with a link to the company's page.
ChatGPT
OAI-SearchBotSearch answers
Finds websites to show in ChatGPT's search answers. Source
ChatGPT-UserPages a person asks for
Opens a page when someone using ChatGPT asks it to. OpenAI says robots.txt rules may not apply to it, because a person asked for the page. Source
GPTBotTraining
Collects pages that may be used to train OpenAI's models. Source
Claude
Perplexity
Common Crawl
CCBotTraining
Common Crawl's open archive of the web, often used to train AI models. Source
Questions
What is robots.txt?
A plain text file at the root of a website, such as yourbusiness.com.au/robots.txt, that asks crawlers to stay out of some or all of the site. It is a request, set out in the robots.txt standard (RFC 9309). It shows what a website asks for, not what a crawler does.
Does blocking GPTBot keep my business out of ChatGPT?
Not by itself. GPTBot collects pages that may be used to train OpenAI's models. ChatGPT's search answers use a different crawler, OAI-SearchBot, and OpenAI says a site that keeps OAI-SearchBot out is not shown in ChatGPT's search answers.
Should I keep AI training crawlers out?
That is your choice to make. Keeping GPTBot, ClaudeBot, Google-Extended and CCBot out asks not to be used for AI training. It does not ask to be left out of AI answers. One thing to know: Google says Google-Extended also covers Gemini using a page to ground an answer.
My robots.txt lets them in. Can AI crawlers still be turned away?
Yes. A firewall or host can turn a crawler away before it reads robots.txt, and Cloudflare can block AI crawlers at its firewall. A check from outside cannot see that, and we never pretend to be an AI company's crawler to find out. When your website is behind Cloudflare, the result says so.
Why does my robots.txt have rules nobody here wrote?
Cloudflare adds a standard block of rules to a website's robots.txt when a setting in its dashboard is switched on, and some website builders write the file for you. The result tells you when your file carries Cloudflare's block. If your builder writes the file, look under its SEO or crawler settings.
What this check is, and is not
It reads one file, once, under our own name, and judges it by the rules of the robots.txt standard. It cannot see a firewall that turns crawlers away by name, and it does not tell you whether AI assistants name your business: a site they can read is not always a site they recommend.
The AI Search Visibility Audit (A$490) checks crawler access along with how the assistants answer: six questions, three times each, in four assistants (72 answers), with a competitor map, the sites the assistants rely on and a prioritised action plan for what to fix, checked by a person before it is sent.
See the audit