# More than 9 in 10 Australian business websites are open to AI training crawlers

Only 1 in 23 ask them to stay out. And of the sites that single out AI, four in five use a rule that came from a Cloudflare setting, not a rule the business wrote. This is Find AI Now's own study of 5,000 Australian business websites.

Find AI Now research. Measured 14 November 2026. Published 17 November 2026.
Page: https://www.findainow.com/research/ai-crawler-scan-2026

## What 5,000 websites say to AI crawlers

A website can ask crawlers to stay away with a small public file called robots.txt. We read that file on a random sample of 5,000 Australian .com.au websites. 3,517 had rules we could read, and every figure below is a share of those.

- **96%** have no rule asking AI training crawlers to stay out. 3,365 of 3,517 sites. Likely range 95% to 96.3%.
- **4.3%** ask at least one AI training crawler to stay out. 152 of 3,517 sites, about 1 in 23. Likely range 3.7% to 5%.
- **82%** of the rules aimed at AI came from one Cloudflare setting. 81 of 99 sites. Likely range 73% to 88%.
- **0.2%** single out a crawler that fetches pages for live AI answers. 7 of 3,517 sites. Likely range 0.1% to 0.4%.

One more: 762 sites (21.7%) have no robots.txt at all. That is fine. No file means no crawler is asked to stay out.

## Three findings

### 1. Most rules aimed at AI came from a setting, not from a rule the business wrote

99 sites are open to crawlers in general and ask an AI crawler to stay out. On 81 of them the rule is a standard block that Cloudflare adds to a site's robots.txt when a setting in its dashboard is switched on. The wording is the same on every one. 17 of the 99 had rules found on no other site in the sample.

On sites that sit behind Cloudflare, 81 of 809 carry that block, about 1 in 10. On sites that do not, a rule aimed at AI is on 13 of 2,708, about 1 in 200.

### 2. The rules are aimed at training, not at answers

AI companies run two kinds of crawler. One kind collects pages that may be used to train future models. The other fetches pages to answer a question someone is asking now, or to build the search index those answers draw on.

Almost every rule we found is for the first kind only. 80 of the 81 Cloudflare blocks ask all four training crawlers to stay out and none of the six answer crawlers. Only 7 sites in 3,517 were open in general and asked an answer crawler to stay out.

So on most of these sites the file asks not to be used for training. It does not ask to be left out of answers. One of the four training names, Google-Extended, also covers Gemini using a page to ground an answer, by Google's own description.

### 3. The sites that turn answer crawlers away turn every crawler away

60 sites (1.7%) ask at least one answer crawler to stay out. 53 of those ask every crawler to stay out, and 41 of them include Google's. 27 of the 53 serve the file a website builder uses for a site that is closed to crawlers.

If one of those is a live business website, it is asking Google to stay out as well, and that is the first thing to fix.

## What to check in five minutes

1. **Open your file** Type your web address followed by `/robots.txt` into a browser, for example `yourbusiness.com.au/robots.txt`. A “not found” page means you have no rules, and no crawler is asked to stay out.
2. **Look for a line that says Disallow: /** A line reading `Disallow: /` with nothing after the slash covers the whole site. The `User-agent` lines just above it name the crawlers it applies to.
3. **If the name above it is *, every crawler is covered** That includes Google. On a live site this is the one to fix first. It can be left over from when the site was being built.
4. **If the names are GPTBot, ClaudeBot, Google-Extended, CCBot** You are asking not to be used for AI training. That is a fair choice. One thing to know: Google says Google-Extended also covers Gemini using a page to ground an answer.
5. **If the names are answer crawlers** OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User: you are asking not to appear in live AI answers. Keep that only if you mean it.
6. **If the file says "BEGIN Cloudflare Managed content"** The rules came from a setting in Cloudflare. Whoever manages your Cloudflare account can see it and change it in the dashboard. Ask them whether it was a decision.

If your website builder writes the file for you, the setting is in the builder, not in the file. Look under its SEO or crawler settings.

## What this does not show

- robots.txt is a request. It shows what a site asks for, not what any crawler does. "Open to" on this page means the file has no rule asking that crawler to stay out of the home page.
- A firewall can turn a crawler away even when robots.txt allows it. We did not test that, and we sent no request under any AI company's name. 252 of the 5,000 sites (5%) refused our own, clearly named request for the file.
- A .com.au address is not always a small business. We removed the 10,000 best connected .com.au sites to lean the sample towards smaller ones, but it still holds some large companies and some dormant sites.
- 1,483 of the 5,000 sampled sites gave us no rules to read: no answer, an error or a refusal. Every percentage here is of the 3,517 that did.
- One file, served for 51 sampled sites from a single server, can be read two ways depending on the software reading it. Read the stricter way, 5.8% ask a training crawler to stay out, not 4.3%, and 94.2% are open. We use the lower count of blocks. The headline holds either way.
- The figures describe this sample on one day. They are not a count of all Australian businesses.

## Method

- **Sample:** 5,000 domains drawn at random from the 1,085,965 .com.au domains in Common Crawl's public web graph for August to October 2026, after removing the 10,000 that other sites link to most. The draw uses a fixed seed, so it can be repeated.
- **Request:** One request per site for /robots.txt, under our own name with a contact address. No page was read and nobody was contacted.
- **Counted as asked to stay out:** The site's home page is disallowed for that crawler, using the matching rules in RFC 9309. A rule that closes one folder is not counted.
- **Training crawlers:** GPTBot, ClaudeBot, Google-Extended, CCBot
- **Answer crawlers:** OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User
- **Ranges:** 95% confidence (Wilson). "Likely range" on this page means that range.
- **Sites named:** None. We publish counts only.
- **Measured:** 14 November 2026, from a data centre in the United States.

## Whose study this is

This is Find AI Now's own research. We chose the sample, wrote the method, ran the measurement on 14 November 2026 and counted the results. Nobody outside Find AI Now paid for it or took part in it. Find AI Now is run by FIND AI NOW PTY LTD in Melbourne.

You may quote these figures with a link to this page. Questions about the method, or a correction: hello@findainow.com. Our other studies are on the [research page](https://www.findainow.com/research).

robots.txt is one part of whether AI assistants can read and name a business. Our [AI Search Visibility Audit](https://www.findainow.com/ai-visibility) checks crawler access along with how the assistants answer for your business.

## Sources

- [Common Crawl: web graphs](https://commoncrawl.org/web-graphs)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)
- [OpenAI: overview of OpenAI crawlers](https://platform.openai.com/docs/bots)
- [Anthropic: does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Google: Google's common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Cloudflare: robots.txt setting (managed robots.txt)](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
