Your Website Has AI Visitors and You're Ignoring Them

Your website got 50 visitors yesterday. At least a few of them weren't human.

I don't mean the spam bots from the early 2000s that used to inflate your hit counter (remember hit counters?). I'm talking about GPTBot, ClaudeBot, PerplexityBot, and a growing list of AI crawlers that are visiting your site, reading your content, and deciding whether you're worth mentioning the next time someone asks an AI a question about your industry.

Most businesses don't know these visitors exist. We didn't, until about six months ago. And when we finally looked, we realised we'd been accidentally blocking some of them for years.

This is the bit where I'd normally say something reassuring like "don't worry, you're not alone." But honestly, the fact that almost everyone is getting this wrong doesn't make it less of a problem. It just means nobody's talking about it yet.

The Invisible Traffic You're Not Measuring

Here's what's actually happening on your website right now, whether your analytics knows it or not.

AI bots account for about 4.2% of all HTML requests globally (Cloudflare Radar, 2025 Year in Review). That might sound small. It's not. When you factor in that overall automated bot traffic now makes up 51% of all web traffic (Imperva Bad Bot Report, 2025), the AI slice is growing faster than anything else.

How fast? GPTBot traffic grew 305% year over year between May 2024 and May 2025 (Cloudflare, 2025). That's not a typo. Googlebot, for comparison, grew 96% in the same period. So AI crawlers are outpacing Google's own bot by more than three to one.

We first noticed this when we were reviewing server logs for a government client late last year. There were user agents in the logs we didn't recognise. Names like "GPTBot/1.2" and "ClaudeBot" and "PerplexityBot." My first thought was, honestly, "is this some new kind of spam?" (Twenty years in web development and my instinct is still to assume the worst about unfamiliar traffic. Old habits.)

It wasn't spam. It was the future of search, casually browsing the site and deciding whether to recommend our client's services to people asking questions on ChatGPT.

Vercel's data tells the same story. AI now accounts for 35.4% of all bot traffic on their network. That's more than a third of all non-human visits. And for Australia specifically, the numbers are even more interesting: 75.8% of AI bot traffic hitting Australian websites comes from ChatGPT-User, OpenAI's search crawler (SEOmator, February 2026).

So Australians are heavy users of AI search. And most Australian businesses have no idea whether those AI tools can even see their websites.

Pretty insane, right?

What These Bots Actually Want

Not all AI crawlers are the same, and lumping them together is a mistake we made at Webcoda for a while. (We've since learned.) Each one has a different job, and understanding what they want changes how you think about them.

GPTBot is OpenAI's crawler. It does two things: it gathers content to train future versions of GPT models, and it powers ChatGPT's search feature. When someone types "best digital agency in Sydney for government websites" into ChatGPT (flattering example, I know), GPTBot's earlier visits determine whether your site gets mentioned. GPTBot made 569 million requests in a single month across Vercel's network alone (The Register, December 2025). That's not a casual browser. That's an industrial-scale reading operation.

ChatGPT-User is different from GPTBot, and this is where it gets confusing. ChatGPT-User is the real-time search bot. When someone asks ChatGPT a question and it goes browsing for current information, that's ChatGPT-User, not GPTBot. You can block one without blocking the other. Whether you should is a different question.

ClaudeBot is Anthropic's crawler. It powers Claude's knowledge base and made 370 million requests in the same month GPTBot hit 569 million (The Register, December 2025). Claude is the model of choice for a lot of enterprise customers right now, so if you're a B2B business, this one matters.

PerplexityBot feeds Perplexity, which is basically an AI-powered answer engine. Perplexity is interesting because it's the one that actually sends traffic back to your site most consistently. Its crawl-to-refer ratio is below 400:1, compared to GPTBot's 3,700:1 (Cloudflare, 2025). In plain English: Perplexity reads 400 pages for every one visitor it sends you. GPTBot reads nearly 4,000.

Google-Extended feeds Google's AI Overviews (the AI-generated answers that appear above normal search results). This is the one that's probably affecting your traffic right now, even if you don't realise it. AI Overviews now appear in 21% of all Google searches globally (Search Engine Journal, November 2025), and the rate is climbing fast. That's more than one in five searches showing an AI answer before the traditional results.

Here's the thing. Each of these bots has its own user agent string, and each one respects (or doesn't respect) its own rules. You can allow GPTBot but block ClaudeBot. You can let PerplexityBot in but keep Google-Extended out. Or you can do what most Australian businesses are doing right now: block them all by accident and not even know it.

How You're Probably Blocking Them (By Accident)

This is the bit that stung when we discovered it at Webcoda. And I'm telling you this because I suspect a lot of agencies and IT teams have made the same mistake.

We were checking a client's robots.txt file as part of a broader SEO review. Pretty routine. But when we looked at the AI crawler section, we found that GPTBot was being blocked by a default rule that had been there since the site launched. Nobody had put it there deliberately. It was part of a boilerplate robots.txt template that shipped with their CMS. The template was written before AI crawlers were a thing, and it had a blanket disallow rule that caught the new bots along with a bunch of older ones nobody cared about.

Then we checked our own site. Same thing. I wish I could tell you we were smarter than our clients, but we weren't. We'd been accidentally blocking some AI crawlers on our own website for months.

And we're not unusual. Roughly a quarter to a third of top websites block GPTBot, depending on which study you look at (Originality.ai, 2023; HTTP Archive, 2025). Among the top 100 news publishers, the rate is much higher: 79% block at least one AI training crawler. Now, some of those publishers are blocking deliberately (they've got good reasons, which I'll get to). But a significant chunk of the blocking is businesses like yours and mine doing it by accident.

Here's how to check yours. It takes about sixty seconds.

Step 1: Open your browser and type your domain followed by /robots.txt. So if your site is example.com.au, go to example.com.au/robots.txt.

Step 2: Look for any of these user agents: GPTBot, ChatGPT-User, ClaudeBot, ClaudeBot-User, PerplexityBot, Google-Extended.

Step 3: If you see Disallow: / under any of those agents, you're blocking them completely. If there's no mention of them at all, they're allowed by default (which is fine, actually).

Step 4: If you see a blanket rule like User-agent: * followed by Disallow: /, that blocks everything, including AI crawlers, Googlebot, and everything else. That's a different kind of problem.

The tricky part is the deliberate vs accidental distinction. The Atlantic has done something genuinely clever here. They've built a scorecard system to evaluate which AI crawlers send traffic back versus which ones just take content (Glenn Gabe, October 2025). Bots that refer visitors get access. Bots that just scrape without sending anyone back get blocked.

That's a strategic decision. What most businesses have isn't a strategy. It's an accident.

Why This Actually Matters Now

I can already hear some of you thinking: "So a few AI bots can't read my website. Why should I care? Google still works fine."

Fair question. Here's the honest answer.

Google's AI Overviews reduce click-through rates by roughly half. Traditional organic search results get about 15% click-through. When an AI Overview appears above them, that drops to 8% (Pew Research Center, 2025). Half your potential clicks, gone, absorbed by the AI answer at the top.

And AI Overviews are showing up in more than one in five Google searches globally, with the rate climbing every quarter. That's not a future scenario. That's happening right now, every day, on your potential customers' screens.

But here's what really got my attention. When someone asks ChatGPT or Perplexity for a recommendation, there's no page 2. There's no ranking position 7 that you can work your way up from. Either you're in the answer, or you don't exist. AI search is binary in a way that Google search never was.

ChatGPT referral traffic increased 52% year over year (Similarweb, December 2025), and ChatGPT drives 87.4% of all AI referral traffic (Conductor, November 2025). That second number is staggering. If you're visible to ChatGPT, you're visible to the AI search world. If you're not, you're basically invisible.

Now, AI referral traffic is still small in absolute terms. About 1.08% of all web traffic (Conductor, November 2025). But McKinsey estimates that AI search will impact $750 billion in revenue by 2028, and the growth trajectory is steep. We've gone from "this doesn't matter" to "this matters" in about eighteen months.

Some publishers are already reporting that AI bots are replacing human traffic on their sites. That's not a comfortable headline for anyone in digital marketing.

And here's the bit that makes it personal for Australian businesses: we're early adopters. Australians have jumped on AI search tools faster than most markets. Which means the impact is going to hit us sooner and harder.

What You Can Actually Do About It

Right, so you've read this far and you're probably somewhere between "slightly concerned" and "mildly panicking." Let me talk you back to "productively concerned." That's the sweet spot.

First: check your robots.txt. I've described how above. This takes a minute. If you're accidentally blocking AI crawlers, fixing it is literally editing a text file. Your developer can do it in five minutes. Don't overthink this. If you're a business that wants to be discovered (and you're not a major news publisher protecting premium content), there's almost no reason to block AI crawlers.

Second: check your server logs. Look for user agents containing "GPTBot," "ClaudeBot," "PerplexityBot," or "ChatGPT-User." This tells you which AI crawlers are already visiting (or trying to visit) your site. If they're there and you're blocking them, that's traffic you're actively turning away.

Third: run an AI accessibility check. This is the bit where I mention that we built a free tool for exactly this. The AI Accessibility Checker scans your website and tells you whether AI crawlers can access your content, whether your robots.txt is configured correctly, and what you're blocking versus what you're allowing. We built it because we wanted to check our own clients' sites after our robots.txt embarrassment, and then we figured other people might find it useful too.

Fourth: make sure your content is actually readable by AI. Having your robots.txt sorted is step one. But if your site is a single-page JavaScript application with no server-side rendering, AI crawlers can't read it even if you invite them in. They've got tight timeouts (1 to 5 seconds) and most won't execute JavaScript. Clean HTML, proper heading structure, and structured data (Schema.org markup) all help AI crawlers understand what your site is about. This isn't fancy. It's the same stuff that helps screen readers and accessibility tools.

Fifth: don't panic, but don't ignore it. This is genuinely new territory. Nobody has all the answers yet, and anyone who tells you they do is selling something. (Yes, I'm aware of the irony.) The key is to stop accidentally blocking AI crawlers and start making deliberate decisions about which ones you want to allow and why. That's a real conversation, not a quick admin decision.

Here's a quick way to check how your website stacks up right now:

The Honest Admission

I've been building websites for twenty years. For about nineteen and a half of those years, the only non-human visitor I cared about was Googlebot. Everything was optimised for Google. Every decision, every technical choice, every piece of metadata.

And then sometime in 2025, the game changed and I didn't notice for months. We had clients whose AI crawlers were being blocked by default configurations we'd set up years ago. We had our own site accidentally turning away the very bots that could have been recommending us to potential customers.

I'm not telling you this to make you feel better about your own situation (though maybe it does, a little). I'm telling you because this is new for everyone. The agencies pretending they've had an "AI SEO strategy" since 2023 are making it up as they go along, same as the rest of us. The difference is whether you acknowledge it and start fixing things, or keep pretending the problem doesn't exist.

Australians are enthusiastic early adopters of AI search. Our potential customers are already asking AI tools for recommendations. And if your website is invisible to those tools, you're losing opportunities you don't even know about.

Check your robots.txt. Run the AI Accessibility Checker. Make deliberate choices instead of accidental ones.

Job done.

---