Add this to your apache configuration (assuming you have mod_rewrite enabled, and assuming you are not using nginx)
RewriteEngine On RewriteCond %{HTTP_USER_AGENT} MJ12bot RewriteRule .* - [F]
Story: This stupid Majestic 12 bot from Britain with a User Agent string of “MJ12” that is totally not a spambot was hammering a Ruby on Rails site on this VPS, causing the system load to shoot up as high as 15.
I have used robots.txt for this kind of thing in the past, but I wanted to make sure. The request is told to go away before it even hits the Rails server. Now, the load is under 1, and the VPS is not using 95% of its swap space.
Much better.
I am also using the WP fail2ban plugin along with a bit of server configuration to stop the constant stream of bots trying to hack into this WordPress site.