Does Hawk Host Block Googlebot or AI Bots?

Hawk Host does not block Googlebot; in fact, it actively allows Googlebot through on all web hosting servers by specifically allowing the Googlebot IP addresses through our Imunify360 web application firewall. We also allow other search engine bots, such as Bingbot, as they provide the IP addresses that Bingbot utilizes. We do, however, initiate rate limiting on specific bots that may bring user sites down without it. There are, however, a variety of third-party services and software that may now be actively blocking Googlebot, Bingbot, and others by default.

Caution

Hawk Host Imunify360 web application firewall may block IP addresses that spoof user agents when the service provides a list of IP addresses it actively uses. We highly recommend against spoofing user agents to test accessibility; instead, rely on the native tools most services provide to test whether they can reach your website.

Cloudflare

As many Hawk Host users use Cloudflare as a CDN or an additional security layer, we believe this information is relevant. If you’re not using Cloudflare, you can safely ignore. Cloudflare has recently announced that on September 15, 2026, it will update its defaults for new domains to block bots classified as training or agents. Hawk Host has received reports and discussions across the wider internet that Googlebot may already be blocked, depending on your Cloudflare settings.

As of September 15, 2026, if you’re using Cloudflare, they will allow you to adjust behaviors based on the type of bot:

• Search: Crawlers that index your website. Examples include typical search engine bots that index your website so it appears in their search results. Bots that act as both training and crawlers would not be considered search bots.

• Agent: Chat fetch bots, such as ChatGPT-User, who visit websites on your behalf. It would also include AI-assisted programming tools that can retrieve documentation in real time.

• Training: Crawlers who specifically crawl websites to train or fine-tune their models. These crawlers may be used not only for training but also for search. Examples would include not only OpenAI and Anthropic, but may also include Googlebot and Bingbot which are dual-purpose bots.

Each of these categories can be configured in your Cloudflare settings to block (on all pages), block on pages with ads (Cloudflare detects pages with ads), or allow (do not block).

You will configure this in Cloudflare’s Security section under Configure AI bot policies.

Hawk Host strongly recommends using robots.txt in most cases to control how known bots scrape your website.

Web Hosting with unrestricted GoogleBot access

Fast NVMe Storage • Free Migrations • 24/7 Expert Support

Explore our Web Hosting Plans