Bad robots can overwhelm your website. Well behaved robots like Google's, will visit your site at a moderate pace, so as not to overwhelm your server and eventually use the data they received from your site to send you visitors. Bad robots usually ignore generally accepted
protocols of conduct and are unlikely to send you visitors. Someone came up with a
scheme to automatically rid your site of bad robots.
The first step is to create a robots.txt file, placed in your root directory (in my case,
http://www.mysqltalk.com/robots.txt)
As per the instructions, it will have the following contents:
User-agent: *
Disallow: /getout.php
Then it asks you to create a directory called "trap", which I've done (
http://www.mysqltalk.com/trap/).
I must note at this point that as mentioned on the site linked to, I have to wait a few days for the good robots to read the new data in robots.txt so that they won't get thrown out along with the bad bots...
Essentially, the next step is to "Chmod your .htaccess file to 644 and chmod getout.php to 755. You should put getout.php in the root folder....SNIP....
Add these lines to your .htaccess file at the very top.
SetEnvIf Request_URI "^(/403.*\.htm¦/robots\.txt)$" allowsome
order deny,allow
deny from env=getout
allow from env=allowsome
"
Then you add
some links to getout.php and use the code as posted
here and you're all set. I'll wait the few days and try it out. Then I'll let you know how it works...
