List of Blocked Bots
Instructions and recommendations from the exon.io team.
At the firewall level, we automatically block selected bots (robots) that unnecessarily load the server or download website content without limits. Some of them also do not respect robots.txt rules. Such behavior can lead to website slowdowns or, in extreme cases, temporary unavailability.
The blocking is configured globally to protect all hosting services. However, if you need a specific bot, it can be allowed for your domain upon request. Tools such as Semrush or Ahrefs can be individually allowed, but we recommend doing so only if you actively use them, otherwise they may unnecessarily increase server load.
We do not block legitimate AI bots in general. Only bots that ignore robots.txt rules or cause excessive server load are blocked.
The meta-externalagent bot was blocked again on August 13, 2026, after being unblocked for three weeks. The bot repeatedly ignores rate limits and HTTP 429 responses, causing excessive load on customer hosting services and website outages. Over the past week, we recorded 15 outages caused by its excessive activity.
|
SEO and scraping bots
Aggressive crawling, high load AhrefsBot AhrefsSiteAudit SemrushBot serpstatbot DotBot MJ12bot BLEXBot SEOkicks DataForSeoBot Timpibot SeekportBot AI and data mining
Massive content scraping Bytespider cohere.ai CCBot meta-externalagent meta-webindexer Amazonbot Security scanners and technical crawlers
Mapping services, technologies, and publicly accessible infrastructure CensysInspect ev-crawler Barkrowler AwarioBot SentiBot |
Scraping frameworks
Automated tools Scrapy heritrix MetadataScraper Regional search engines
Low value vs. load Baiduspider YandexBot Sogou PetalBot Other bots
Tracking and utility services CriteoBot comscore Usercentricsbot Clarity-Bot Brightbot Orbbot IbouBot Dataprovider CMS-Checker CacheWarmerBot carbon umbrella fidget-spinner-bot test-bot Turnitin |
Was this answer helpful?
Your response helps us improve our guides.