So this is just to ban non-targeted crawlers? Any particular reason you'd want to ban crawlers from your site? Surely your server is up to the task of serving a few extra requests, enough so that it's not worth your time adding code (and slowing down good requests) to restrict them.
The kind of bot that I care about are the ones that spam up my content site. They only go to pages that real users visit (the "Post Stuff" page), so this trick wouldn't help against them. And they never post from the same IP twice, preferring to hop between infected machines on a botnet every time they make a post.
I'm curious what sort of traffic pattern this author is seeing that would motivate him to build this.
The kind of bot that I care about are the ones that spam up my content site. They only go to pages that real users visit (the "Post Stuff" page), so this trick wouldn't help against them. And they never post from the same IP twice, preferring to hop between infected machines on a botnet every time they make a post.
I'm curious what sort of traffic pattern this author is seeing that would motivate him to build this.