DragonFly On-Line Manual Pages

EXCLUDE_ROBOTS(1)     User Contributed Perl Documentation    EXCLUDE_ROBOTS(1)

NAME
       exclude_robot.pl - a simple filter script to filter robots out of
       logfiles

SYNOPSIS
           exclude_robot.pl
               -url <robot exclusions URL>
               [ -exclusions_file <exclusions file> ]
               <httpd log file>

           OR

           cat <httpd log file> | exclude_robot.pl -url <robot exclusions URL>

DESCRIPTION
       This script filters HTTP log files to exclude entries that correspond
       to know webbots, spiders, and other undesirables.  The script requires
       a URL as a command line option which should point to a text file
       containing a linebreak separated list of lowercase strings to match on
       for bots. This is based on the format used by ABC
       (<http://www.abc.org.uk/exclusionss/exclude.html>).

       The script filters httpd logfile entries either from a filename
       specified on the command line, or from STDIN. It outputs filtered
       entries to STDOUT.

OPTIONS
       -url <robot exclusions URL>
           Specify the URL of file to grab which contains the list of agents
           to exclude.  The option is REQUIRED.

       -exclusions_file <exclusions file>
           Specify a file to save excluded entries from the logfile. This
           option is OPTIONAL.

AUTHOR
       Ave Wrigley <Ave.Wrigley@itn.co.uk>

COPYRIGHT
       Copyright (c) 2001 Ave Wrigley. All rights reserved. This program is
       free software; you can redistribute it and/or modify it under the same
       terms as Perl itself.

POD ERRORS
       Hey! The above document had some coding errors, which are explained
       below:

       Around line 100:
           You forgot a '=back' before '=head1'

perl v5.20.2                      2001-05-25                 EXCLUDE_ROBOTS(1)