Blocking user agents, bots, crawlers, and scrapers via .htaccess

Your website’s .htaccess file can be used to block specific user agents. This is useful when dealing with bots, scrapers, or crawlers performing unwanted activity on your site.

Identifying user agents in access logs

Common examples of user agent strings as they appear in your access logs:

"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"

"meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"

"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

"Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"

Blocking a single user agent

Add the following to your site’s .htaccess file, replacing UserAgentName with the agent string from your logs:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} UserAgentName [NC]
RewriteRule .* - [F,L]

Blocking multiple user agents

Add the following to your site’s .htaccess file, replacing the agent names with those from your logs:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(UserAgent1|UserAgent2|UserAgent3).*$ [NC]
RewriteRule .* - [F,L]

Common user agent names

User Agent Name	String
Googlebot	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
bingbot	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Slurp	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Baiduspider	Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
YandexBot	Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
AhrefsBot	Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
SemrushBot	Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
MJ12Bot	MJ12bot/v1.4.0 (http://www.majestic12.co.uk/bot.php?+)
facebookexternalhit	facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Twitterbot	Twitterbot/1.0
LinkedInBot	LinkedInBot/1.0 (compatible; Mozilla/5.0; +http://www.linkedin.com)
Slackbot	Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
GPTBot	GPTBot/1.0 (+https://openai.com/gptbot)
ChatGPT-User	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-SearchBot	Mozilla/5.0 (compatible; OAI-SearchBot +https://www.openarchives.org/Register/BrowseSites)
ClaudeBot	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
anthropic-ai	Mozilla/5.0 (compatible; anthropic-ai/1.0; +http://www.anthropic.com/bot.html)
PerplexityBot	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://docs.perplexity.ai/docs/perplexity-bot)
Sogou web spider	Sogou web spider/4.0 (+http://www.sogou.com/docs/help/webmasters.htm#07)
PetalBot	Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)