I'm not sure if this is the right forum for this. We just got hammered by crawler 80legs.com. They came from multiple IP-Addresses simultaneously and hammered 2 sites of ours quite hard.
The useragent-string seems to stay the same:
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html
I blocked them via .htaccess but was wondering. Would the recently added search-engine/crawler-protection help with this? I haven't turned it on yet, but plan to do so soon.
If it doesn't get blocked, can you add it or can I do this myself server-wide?