Skip to main content
Article

My small business site is being hammered by bots: how do I stop them without blocking Google or ChatGPT?

How to stop bot traffic on your website: find which bots hit you, verify Googlebot, pick the right Cloudflare setting, and keep AI search crawlers in.

Ramin MoradiWritten by Published Updated 8 min read
Under a desk lamp, a monitor shows a traffic chart with one sharp red spike above a list of rows tagged green, amber and red, while a hand crosses out one row on a printed checklist next to a phone showing a shop's product page.
Contents

Find out which bots are hitting which pages before you block anything. Then challenge or rate-limit the abusive traffic, check that anything calling itself Googlebot really is Google, and keep allowing the search and AI search crawlers you want to be found by. Robots.txt only steers well-behaved crawlers, so it won't stop the bots causing the trouble.

That is the decision this page is for. Bot-protection vendor pages sell the switch; they rarely ask which crawlers you want to keep. This page sits under website maintenance. For what ongoing care of a site costs, see website maintenance cost. If being cited by ChatGPT matters to you, read how to rank in ChatGPT before you block any AI crawler.

Which bots are hitting my site, and which ones should I keep?

Most bot traffic falls into a few groups, and only some of them are a problem. Search crawlers such as Googlebot and Bingbot put you in search results. AI search crawlers put you in AI answers. Training crawlers collect pages for model training. Then there are scrapers, form spammers and login guessers, which are the ones worth stopping.

OpenAI's own crawler page shows why the names matter. OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features", and sites that opt out of it "will not be shown in ChatGPT search answers". GPTBot is the training crawler, and ChatGPT-User is a fetch that a person triggers from inside ChatGPT (OpenAI, Overview of OpenAI crawlers). OpenAI says each setting is independent of the others.

Start with the data you already have. Your host's access logs, or the bot and traffic analytics in Cloudflare if you use it, will show which user agents and IP ranges send the most requests and which pages they hit. A login page or a contact form taking hundreds of hits from one network is a different problem from a crawler reading every blog post.

Will robots.txt stop bad bots?

No. Robots.txt is a request, and only polite crawlers follow it. Google's own guide says: "The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them" (Google Search Central, robots.txt introduction).

The same guide says robots.txt is used "mainly to avoid overloading your site with requests", and it is good at that for crawlers that listen. Cloudflare makes the same point about AI crawlers: robots.txt "directives are voluntary. Crawlers can ignore them" (Cloudflare, AI Crawler and Agent access policies). A bot that is guessing passwords on your login page has already decided not to listen, so it needs a firewall rule, a challenge or a rate limit instead.

How do I check that a visitor claiming to be Googlebot really is Google?

Run a reverse DNS lookup on the IP address in your logs, then a forward lookup on the name it returns. Google's guide says a genuine Googlebot address resolves to a name under googlebot.com, google.com or googleusercontent.com, and the forward lookup must return the same IP (Google, verifying Google crawlers).

Anyone can type "Googlebot" into a user-agent string, and Google wrote the guide for site owners worried about "spammers or other troublemakers" claiming to be Google. A real Googlebot also gives itself away by its behaviour. Google says its common crawlers such as Googlebot "always respect robots.txt rules for automatic crawls", so a "Googlebot" hammering a path your robots.txt disallows is worth checking first.

Google also publishes its crawler IP ranges as JSON files, which is the easier route if you want to allow or block by address at scale.

Which Cloudflare setting should I turn on?

If you are on Cloudflare, the free Bot Fight Mode is the blunt option and Super Bot Fight Mode on paid plans is the adjustable one. Bot Fight Mode "Cannot be customized, adjusted, or reconfigured via WAF custom rules", and Cloudflare warns that it "may challenge API or mobile app traffic" (Cloudflare, Bot Fight Mode).

Super Bot Fight Mode is "included in your Pro, Business, or Enterprise subscription" and adds "configurable actions per bot category, bot analytics, and the ability to create exceptions" (Cloudflare, Super Bot Fight Mode). That matters for the question in the title. You can challenge definitely automated traffic while leaving the Verified bots category, which is where Cloudflare puts honest crawlers such as search engines, set to allow.

Cloudflare's AI Crawler and Agent access policies split AI traffic into three behaviours you can treat differently: Search, Agent and Training. Cloudflare says "Training actions also apply to mixed-purpose crawlers used for both Search and Training", so a "block training" setting can catch more than you meant. Read the list before you save it.

Under Attack mode is a different tool. Cloudflare calls it "one of the last resorts when a zone is under attack" and says it "will temporarily pause access to your site and impact your site analytics" (Cloudflare, Under Attack mode). If your site only stays up with it switched on, the fix is a targeted rule for the traffic that is hurting you, not leaving the emergency switch on.

Can I block AI training bots and still appear in ChatGPT or Google's AI answers?

Mostly, yes, because the training and search crawlers use different names. OpenAI says webmasters can allow OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot to keep content out of training (OpenAI crawlers). On Google, the AI answers in Search draw on the normal search index, and Gemini training has its own robots.txt token.

For Google, a page must be indexed and eligible to show in Search with a snippet to appear as a supporting link in AI Overviews or AI Mode, so Googlebot access is what counts there (Google, AI features and your website).

Google-Extended is the robots.txt token for Gemini. Google says it controls whether content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google common crawlers). Blocking it keeps you in Google Search, but your content can no longer be used for grounding in Gemini Apps.

Whether to allow training at all is a business choice. If you want AI tools to know your business when people ask about your field, blocking training crawlers works against that.

What do we allow on overlord.team?

We allow every search and AI crawler we know of, and keep private paths closed to all of them. Our live robots.txt, checked on 12 November 2026, names 50 user agents in one group, from GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended to Googlebot, bingbot and the link-preview bots for LinkedIn, Slack and WhatsApp.

That group gets exactly the same rules as every other visitor. Admin, login, API, booking-management and internal search paths are disallowed, as are URLs carrying tracking parameters such as utm_, gclid and fbclid, which would otherwise create extra URLs for the same page. We want to be cited, so we let training crawlers in as well. That is our choice for a site that sells expertise, and a shop with a paid catalogue might reasonably choose differently.

The site sits behind Cloudflare. We don't publish our firewall or bot rules, for the obvious reason.

When should I hand bot trouble to someone else?

When it keeps coming back, when it targets your login or checkout, or when nobody on your team can read the logs. A one-off crawl spike can be handled with a rule. A recurring attack on a login form is a maintenance and monitoring job, because it needs someone watching the site after the rule goes in.

Our website maintenance retainer starts from $350 per site or app, per month, and covers patches, updates, backup checks with a restore test, and an uptime and certificate watch if you don't already buy monitoring. Anything beyond that list, such as bot and firewall rules for a specific site, is quoted after a written scope.

FAQ

Should I block GPTBot?

Only if you don't want your pages used to train OpenAI's models. Blocking GPTBot doesn't remove you from ChatGPT search answers, because OpenAI uses OAI-SearchBot for search and says the two settings are independent. If you want to be found in ChatGPT, keep OAI-SearchBot allowed whatever you decide about GPTBot.

Will blocking bad bots improve my Google rankings?

Not directly. Google doesn't rank you higher for blocking bots. Blocking abusive traffic can keep the site fast and online for real visitors and for Googlebot, though, and Google's robots.txt guide notes that crawl rules exist "mainly to avoid overloading your site with requests".

A drop in sessions can also come from a Search Console change, an analytics tag or a Google update, not just bots. See what to check when website traffic dropped for those checks.

Can I just block every bot I don't recognise?

You can, but you will probably block something you wanted, such as a link-preview bot or an uptime monitor. Allow verified crawlers first, then challenge or rate-limit the rest by behaviour. On Cloudflare, check the challenged requests in your security analytics for a week before you switch from challenge to block.

Is Cloudflare's free plan enough to stop bots?

It can be, with limits. Bot Fight Mode is free but can't be adjusted with custom rules and may challenge API or app traffic. If you need exceptions or per-category actions, Super Bot Fight Mode needs a Pro plan or higher. A badly configured block can cost you more traffic than the bots did.

What to do next

Pull a week of your logs or bot analytics and list the top user agents by request count. If you would like help reading that list and deciding what to allow, book a free 30-minute assessment.

How this guide was made: Ramin Moradi researched and wrote it with AI assistance. Google, OpenAI and Cloudflare documentation and our own robots.txt were checked against the linked pages on 12 November 2026 (Europe/Istanbul). Vendor settings and crawler names change often, so check the current docs before you change a rule.

Comments

Loading comments

Ready to get started?

If this page matches the job, send the brief. We will say no if it does not.