Skip to content
Hawk Host Knowledge Base

Blocking user agents, bots, crawlers, and scrapers via .htaccess

Your website’s .htaccess file can be used to block specific user agents. This is useful when dealing with bots, scrapers, or crawlers performing unwanted activity on your site.

Identifying user agents in access logs

Section titled “Identifying user agents in access logs”

Common examples of user agent strings as they appear in your access logs:

"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
"meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
"Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"

Blocking a single user agent

Section titled “Blocking a single user agent”

Add the following to your site’s .htaccess file, replacing UserAgentName with the agent string from your logs:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} UserAgentName [NC]
RewriteRule .* - [F,L]

Blocking multiple user agents

Section titled “Blocking multiple user agents”

Add the following to your site’s .htaccess file, replacing the agent names with those from your logs:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(UserAgent1|UserAgent2|UserAgent3).*$ [NC]
RewriteRule .* - [F,L]

Common user agent names

Section titled “Common user agent names”
User Agent NameString
GooglebotMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
bingbotMozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
SlurpMozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
BaiduspiderMozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
YandexBotMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
AhrefsBotMozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
SemrushBotMozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
MJ12BotMJ12bot/v1.4.0 (http://www.majestic12.co.uk/bot.php?+)
facebookexternalhitfacebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
TwitterbotTwitterbot/1.0
LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; +http://www.linkedin.com)
SlackbotSlackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
GPTBotGPTBot/1.0 (+https://openai.com/gptbot)
ChatGPT-UserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-SearchBotMozilla/5.0 (compatible; OAI-SearchBot +https://www.openarchives.org/Register/BrowseSites)
ClaudeBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
anthropic-aiMozilla/5.0 (compatible; anthropic-ai/1.0; +http://www.anthropic.com/bot.html)
PerplexityBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://docs.perplexity.ai/docs/perplexity-bot)
Sogou web spiderSogou web spider/4.0 (+http://www.sogou.com/docs/help/webmasters.htm#07)
PetalBotMozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)