Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 2 down vote favorite

How do I realize a UA string block by regular expression in the config files of my Apache webserver?

For example: if I would like to block out all bots from Apache on my debian server, that have the regular expression /\b\w+[Bb]ot\b/ or /Spider/ in their user-agent.

Those bots should not be able to see any page on my server and they should not appear neither in the accesslogs nor in the errorlogs.

http://global-security.blogspot.de/2009/06/how-to-block-robots-before-they-hit.html supposes to uses mod_security for that, but isn't there a simple directive for http.conf?

share|improve this question
Are you sure you don't want to just have them not crawl your site using a robots.txt? (seems more efficient then doing an Apache regex for every request) – dan Jun 26 at 5:53
I have many websites on the server, so I would have to add a robots.txt to each folders at once and only if there is none yet. the simplest solution would be such a directive in the main config. And I want to unclutter the accesslogs and errorlogs too – rubo77 Jun 26 at 6:00
I see. I'm not sure specific regexes are going to catch them all, particularly those with differing or missing UA's. See this for more: Crawler Identification and Examples – dan Jun 26 at 6:12
I know I can't get them all, but I still would like to add this easy solution to reduce some traffic and logging. I dot't need a complete solution, just an easy question: What line would I have to add in the apache-config? I can't find it ;) – rubo77 Jun 26 at 6:29
1  
I'll take a look at the regex for what you have. Here's list of UA's for spiders, robots, and crawlers though: List of User-Agents The regex will be very incomplete indeed. – dan Jun 26 at 6:31
show 2 more comments

1 Answer

up vote 1 down vote

I enabled rewrite engine in Apache:

a2enmod rewrite

and added this block to my /etc/apache2/httpd.conf 

<Directory /var/www/>
       <IfModule mod_rewrite.c>
                RewriteEngine on
                RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR]
                RewriteCond %{HTTP_USER_AGENT} sosospider [NC,OR]
                RewriteCond %{HTTP_USER_AGENT} BaiduSpider [NC]
                # Allow access to robots.txt and forbidden message
                # at least 403 or else it will loop
                RewriteCond %{REQUEST_URI} !^/robots\.txt$
                RewriteCond %{REQUEST_URI} !^/403\.shtml$
                RewriteRule ^.* - [F,L]
       </IfModule>
</Directory>

and restarted Apache:

apache2ctl graceful

now these calls from those spiders all cause 403 Errors:

grep -E 'spider|bot' /var/log/apache2/*.log
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.