Tomfoolery Logging Bot hits

[jdmfarms]

Hello.

As I'm new, I'm still learning what I can about userspice. I haven't found anything that addresses this subject.

Issue: I've noticed in my log a constant logging of certain IPs and did some research and noticed they are the search engine web crawlers. I'd like the log to omit entries of known bots (google, amazon, etc.)

Thoughts: I was thinking of exploring the tomfool php file and support files to see if I could put an exception in but before hand, was interested in seeing if any others have done this already. Another thought is to explore code in the htaccess files in select directories to block bots from scanning those directories.

Your thoughts? Thank you for any insight you can provide on this. I just dont want my log bloated with bot hits.

JDM

[mudmin]

We can definitely make a whitelist. I actually don't have that problem on my projects, so it could be some sort of server configuration option.

My thought is that we can add a table of ip addresses that don't show up there. Just make sure not to put your own!

It's going to be 5 weeks until I can get back to active development (I run a camp and we're right in the middle of camp season), but this is doable.

[Brandin]

Very odd that you are getting forked with these crawlers! I haven't had any issues on either of my US projects...However, if it is happening, adding an omitting option would be great.

[jdmfarms]

I'm an amateur programmer...still have lots to learn. Therefore, it could be something I could turn off on my server. Or, it could be that I'm on a European service provider's server.

[Brandin]

I don't have any special configuration on my cPanel VPS other than ConfigServe firewall...I am wondering if this is why I don't get crawled...

[firestorm]

you generaly do this in a robot.txt file however if that fails to stop them then you can use the following in your htaccess for the directory you dont want crawled

<pre>

Code:

RewriteEngine On

RewriteCond {3bc1fe685386cc4c3ab89a3f76566d8931e181ad17f08aed9ad73b30bf28114d}{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]

</pre>


change the bots to the ones you want stopped and they arent case sensitive, you can change the error code to what you wish aswell, currently 403 access forbidden

[jdmfarms]

Firestorm, unfortunately neither seems to work. I tried the htaccess first and it crashed my site. now I have the robot.txt file in a few folders and they seem to ignore it. however, i only have the basic entry:

User-agent: *
Disallow: /