Fail2stat

The Idea

I will explain how to combine your favorite website statistics software with fail2ban in order to produce statistics for break-in attempts at your server. The idea is simple: let fail2ban convert any ban triggering line in a log file to a web server log file format; let the newly created pseudo webserver log file get examined by your website statistics software. Here I treat the special case of an Apache web servers running with Webalizer.

The converter

Adapt the following script to your needs and save it, executable by the fail2ban user, in /usr/local/bin/fail2stat.

#!/bin/bash
# fail2stat
#       makes fail2ban events processable for webalizer
#
IP="$1"
NAME="$2"
LOGPATH="$3"
TMP="/tmp/fail2stat$(date +%s)" # tmp file distinguishing seconds for handling rapid attacks
# get the log lines
grep $IP $LOGPATH > $TMP
# don't process if they are from an apache log file
if [[ "$LOGPATH" == *"apache"* ]]; then cat $TMP >> /var/log/fail2stat.log; rm /tmp/fail2stat; exit 0; fi
# processing in the following way is valid for auth.log and mail.log; 
# adapt if you need differently formatted logs
while read MONTH DAY TIME TARGET DAEMON ETC;
        do
                # as the year is not part of the log line, we have to take 
                # extra care around New Year.
                YEAR="$(date +%Y)"
                # Adapt the date command to suit your locale.
                MONTH_NUMBER="$(date -d "$MONTH 1, $YEAR" +%m)"
                if [[ $((MONTH_NUMBER-1)) -ge $(date +%m) ]]; then YEAR=$((YEAR -1)); fi
                #
                # add zeroes to fist nine days of month
                if [[ 9 -ge $DAY ]]; then DAY="0$DAY"; fi
                #
                DAEMON="${DAEMON%%[*}"
                MSG="${ETC#*: }"; MSG="${MSG%%from *}"
                MSG="$(echo $MSG | sed 's/ /_/g')"
                USER="${MSG##*user?}"; USER="${USER%%_*}";
                # the numbers 200 and 1024 are arbitrary
                echo "$IP - \"$USER\" [$DAY/$MONTH/$YEAR:$TIME +0000] \"GET /$NAME/$MSG/ HTTP/1.1\" 200 1024 \"-\" \"$TARGET/$DAEMON\"" >> /var/log/fail2stat.log
        done < $TMP
rm $TMP
exit 0

logrotation

Create the following file as /etc/logrotate.de/fail2stat.

/var/log/fail2stat.log {

    daily
    rotate 4
    compress

    delaycompress
    missingok

    # If fail2ban runs as non-root it still needs to have write access
    # to logfiles.
    # create 640 fail2ban adm
    create 640 root adm
}

This rotates the log file daily, up to three older versions and leaves the first older version fail2stat.log.1 uncompressed. This is important since webalizer will work on this one.

fail2ban configuration

jail.local

Append the following lines to /etc/fail2ban/jail.local under [Default] > ACTIONS. First, define the statistics action

# statistics action
stat = fail2stat

Append the statistics action at your favorite action shortcut, e.g. create

# ban & send & log stats
action_mwls = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
               %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
               %(stat)s[name=%(__name__)s,logpath="%(logpath)s"]

and, finally, set the default action to include statistics:

action = %(action_mwls)s

If you have defined jails with individual actions, take care to include the statistics action there as well.

action.d/fail2stat.conf

Copy a standard configuration file and change the line defining actionban to

actionban = /usr/local/bin/fail2stat <ip> <name> <logpath>

The complete file might look like this (to be uploaded).

Now it is time to restart fail2ban.

apache configuration

Set up a new virtual host. The configuration in /etc/apache2/sites-available/fail2stat.conf might be like this:

<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        ServerName yourserver.local
        DocumentRoot /var/www/fail2stat
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>
        <Directory /var/www/fail2stat>
                Options Indexes FollowSymLinks MultiViews
                #AllowOverride None
                AllowOverride All
                Order allow,deny
                allow from all
        </Directory>
</VirtualHost>

Of course, I would recommend to run the site with https. Manuals for doing this can easily be found elsewhere.

The ServerName depends on whether you want to access the server from the internet or not. If so, you have to take care of DNS settings as well which is beyond the scope of this article.

Make sure that the DocumenRoot path exists and is writeble by the webalizer user.

Finally, activate the new site by executing with root privileges

a2ensite /etc/apache2/sites-available/fail2stat.conf
systemctl restart apache2

In the last line reload would be enough if the webserver is already running.

webalizer configuration

I assume you have webalizer already running as a cronjob. In this case just copy the configuration

cp /etc/webalizer/webalizer.conf /etc/webalizer/fail2stat.conf

and edit it to match the following lines:

LogFile /var/log/fail2stat.log.1
OutputDir /var/www/fail2stat
ReportTitle Fail2Ban statistics for
CacheIPs yes
CacheTTL 30
GeoDB yes
TopKSites 0
TopKURLs 0
TopReferrers 0
TopEntry 0
TopExit 0
TopSearch 0
AllSites yes
AllURLs yes
AllAgents yes
AllUsers yes

The OutputDir must match the DocumentRoot of the apache configuration.

The two "Cache" lines make sure that unresolvable IPs are not tried to be resolved all the time, the next time only after 30 days. This is useful because a lot of the devices trying to break into your system are not resolvable.

The use of GeoDB is strongly recommended if you want to have a country diagram without the break-in attempts reducing your bandwith any further. Apart from choosing "yes" in the config file you have to make sure that the latest GeoDB is installed at the path given by GeoDBDatabase.

The 0s at the "Top" entries disable these data; they have no meaning in the converter.

The "All" parameters ensure that you do not miss any break-in attempt in the statistics display.

Test your configuration by

webalizer -c /etc/webalizer/fail2stat.conf

, maybe with root privileges, depending on your webalizer installation.

That's It

Now you should be able to see some useful statistics of fail2ban visiting http://yourserver.local.