Skip to content

Find AI Now research · measured 14 November 2026

More than 9 in 10 Australian business websites are open to AI training crawlers

Only 1 in 23 ask them to stay out. And of the sites that single out AI, four in five use a rule that came from a Cloudflare setting, not a rule the business wrote. This is Find AI Now's own study of 5,000 Australian business websites.

The numbers

What 5,000 websites say to AI crawlers

A website can ask crawlers to stay away with a small public file called robots.txt. We read that file on a random sample of 5,000 Australian .com.au websites. 3,517 had rules we could read, and every figure below is a share of those.

  • 96%

    have no rule asking AI training crawlers to stay out

    3,365 of 3,517 sites. Likely range 95% to 96.3%.

  • 4.3%

    ask at least one AI training crawler to stay out

    152 of 3,517 sites, about 1 in 23. Likely range 3.7% to 5%.

  • 82%

    of the rules aimed at AI came from one Cloudflare setting

    81 of 99 sites. Likely range 73% to 88%.

  • 0.2%

    single out a crawler that fetches pages for live AI answers

    7 of 3,517 sites. Likely range 0.1% to 0.4%.

One more: 762 sites (21.7%) have no robots.txt at all. That is fine. No file means no crawler is asked to stay out.

What we found

Three findings

1. Most rules aimed at AI came from a setting, not from a rule the business wrote

99 sites are open to crawlers in general and ask an AI crawler to stay out. On 81 of them the rule is a standard block that Cloudflare adds to a site's robots.txt when a setting in its dashboard is switched on. The wording is the same on every one. 17 of the 99 had rules found on no other site in the sample.

On sites that sit behind Cloudflare, 81 of 809 carry that block, about 1 in 10. On sites that do not, a rule aimed at AI is on 13 of 2,708, about 1 in 200.

2. The rules are aimed at training, not at answers

AI companies run two kinds of crawler. One kind collects pages that may be used to train future models. The other fetches pages to answer a question someone is asking now, or to build the search index those answers draw on.

Almost every rule we found is for the first kind only. 80 of the 81 Cloudflare blocks ask all four training crawlers to stay out and none of the six answer crawlers. Only 7 sites in 3,517 were open in general and asked an answer crawler to stay out.

So on most of these sites the file asks not to be used for training. It does not ask to be left out of answers. One of the four training names, Google-Extended, also covers Gemini using a page to ground an answer, by Google's own description.

3. The sites that turn answer crawlers away turn every crawler away

60 sites (1.7%) ask at least one answer crawler to stay out. 53 of those ask every crawler to stay out, and 41 of them include Google's. 27 of the 53 serve the file a website builder uses for a site that is closed to crawlers.

If one of those is a live business website, it is asking Google to stay out as well, and that is the first thing to fix.

For business owners

What to check in five minutes

  1. Open your file

    Type your web address followed by /robots.txt into a browser, for example yourbusiness.com.au/robots.txt. A “not found” page means you have no rules, and no crawler is asked to stay out.

  2. Look for a line that says Disallow: /

    A line reading Disallow: / with nothing after the slash covers the whole site. The User-agent lines just above it name the crawlers it applies to.

  3. If the name above it is *, every crawler is covered

    That includes Google. On a live site this is the one to fix first. It can be left over from when the site was being built.

  4. If the names are GPTBot, ClaudeBot, Google-Extended, CCBot

    You are asking not to be used for AI training. That is a fair choice. One thing to know: Google says Google-Extended also covers Gemini using a page to ground an answer.

  5. If the names are answer crawlers

    OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User: you are asking not to appear in live AI answers. Keep that only if you mean it.

  6. If the file says "BEGIN Cloudflare Managed content"

    The rules came from a setting in Cloudflare. Whoever manages your Cloudflare account can see it and change it in the dashboard. Ask them whether it was a decision.

If your website builder writes the file for you, the setting is in the builder, not in the file. Look under its SEO or crawler settings.

Being straight

What this does not show

  • robots.txt is a request. It shows what a site asks for, not what any crawler does. "Open to" on this page means the file has no rule asking that crawler to stay out of the home page.
  • A firewall can turn a crawler away even when robots.txt allows it. We did not test that, and we sent no request under any AI company's name. 252 of the 5,000 sites (5%) refused our own, clearly named request for the file.
  • A .com.au address is not always a small business. We removed the 10,000 best connected .com.au sites to lean the sample towards smaller ones, but it still holds some large companies and some dormant sites.
  • 1,483 of the 5,000 sampled sites gave us no rules to read: no answer, an error or a refusal. Every percentage here is of the 3,517 that did.
  • One file, served for 51 sampled sites from a single server, can be read two ways depending on the software reading it. Read the stricter way, 5.8% ask a training crawler to stay out, not 4.3%, and 94.2% are open. We use the lower count of blocks. The headline holds either way.
  • The figures describe this sample on one day. They are not a count of all Australian businesses.

How we measured

Method

Sample
5,000 domains drawn at random from the 1,085,965 .com.au domains in Common Crawl's public web graph for August to October 2026, after removing the 10,000 that other sites link to most. The draw uses a fixed seed, so it can be repeated.
Request
One request per site for /robots.txt, under our own name with a contact address. No page was read and nobody was contacted.
Counted as asked to stay out
The site's home page is disallowed for that crawler, using the matching rules in RFC 9309. A rule that closes one folder is not counted.
Training crawlers
GPTBot, ClaudeBot, Google-Extended, CCBot
Answer crawlers
OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User
Ranges
95% confidence (Wilson). "Likely range" on this page means that range.
Sites named
None. We publish counts only.
Measured
14 November 2026, from a data centre in the United States.

Whose study this is

This is Find AI Now's own research. We chose the sample, wrote the method, ran the measurement on 14 November 2026 and counted the results. Nobody outside Find AI Now paid for it or took part in it. Find AI Now is run by FIND AI NOW PTY LTD in Melbourne.

You may quote these figures with a link to this page. Questions about the method, or a correction: hello@findainow.com. Our other studies are on the research page.

robots.txt is one part of whether AI assistants can read and name a business. Our AI Search Visibility Audit checks crawler access along with how the assistants answer for your business.

Sources

Reusing the figures

The figures are published under the Creative Commons Attribution 4.0 licence: reuse them with credit to Find AI Now and a link to this page. The figures as data (JSON) · This study as Markdown