Twiceler Bot is Garbage

Every few months or so we end up having to block the Twiceler bot on one of our servers due to high traffic caused by it resulting in what almost could be described as a mini dos attack.  The bot is made by Cuill which supposedly has some people from Google working on it as well as quite a bit of venture capital.  It sure does not show with their Twiceler bot however.  It is supposedly experimental at this point but they’ve let it loose on the Internet and every web master out there is complaining about it.

I’ve read about people complaining about it using several GB of bandwidth in a short amount of time and visiting close to 100,000 pages with most of them being the same pages.  So it hits a site using say a gallery system and all hell breaks loose and it just causes a slew of problems.

So we’ve pretty much blocked all their IP ranges like everyone else out there in hopes that maybe they’ll fix their bot to maybe read robots.txt files and also crawl at a reasonable speed.