In my last post we installed Logwatch as well as OSSEC. It is now time to get Logwatch and OSSEC playing together in the same sandbox. In this post I’ll discuss how to get Logwatch to summarize the information being generated by OSSEC.
Deployment Options
We have two paths we can follow to set this up:
- Have Logwatch parse the OSSEC logs directly.
- Have OSSEC send its alerts to a Syslog type server, then run Logwatch on the Syslog server.
The benefit to option #1 is that we only need one system. Logwatch will be run on the system hosting the OSSEC server. The problem we are going to run into however involves the OSSEC alert file. Log entries do not get normalized. This means the format can change from entry to entry, and may even spread over multiple lines. It is going to be a real nightmare to create a Logwatch script that will filter and summarize the alert information.
If we go with option #2, we will require another box to act as our centralized logging server. In order to have the OSSEC server accept log entries from non-agent systems, it has to listen on UDP/514. This is the same port used by a centralized logging server, and you can’t have two applications share the same port (except with Windows, but socket access is extremely messy). On the plus side, the alert entries will get normalized when they are transmitted to the Syslog server so creating a Logwatch summary script will be far easier. Further, Logwatch already knows about the standard Syslog files, so we will have less customization work to do.
Finally, I mentioned in an earlier post that OSSEC is not designed to be a SIM. This is because it does not record everything, just the events that generate an alert. So we are probably going to want a centralized server anyway, and it makes sense to have it store the information being generated by OSSEC.
So if it sounds like I am steering you towards option #2, you are absolutely correct. With that said, I’m actually going to cover option #1 as it is a far more complex setup.
Dealing With Date/Time Stamps
Have a look at the main OSSEC logfile and you should see similar to the following:
[root@fubar logs]# tail -3 /var/ossec/logs/ossec.log
2010/02/18 12:32:05 ossec-rootcheck: INFO: Ending rootcheck scan.
2010/02/18 14:27:06 ossec-syscheckd: INFO: Starting syscheck scan.
2010/02/18 14:39:21 ossec-syscheckd: INFO: Ending syscheck scan.
Note the way the date/time stamp is formatted. This is different than most applications, so the first thing we will need to do is tell Logwatch how to deal with this format. We will need to create a script that we can call when needed that understands the format shown above.
Start by moving into the shared scripts directory:
cd /usr/share/logwatch/scripts/shared
Using your favorite editor, create a file named “applylongdate”:
vi applylongdate
Here is what you need inside of that file. Feel free to copy/paste from this page:
use Logwatch ‘:dates’;
my $Debug = $ENV{‘LOGWATCH_DEBUG’} || 0;
$SearchDate = TimeFilter(‘%Y/%m/%d %H:%M:%S’);
if ( $Debug > 5 ) {
print STDERR “DEBUG: Inside ApplyLongDate…\n”;
print STDERR “DEBUG: Looking For: ” . $SearchDate . “\n”;
}
while (defined($ThisLine = <STDIN>)) {
if ($ThisLine =~ m/^$SearchDate /o) {
print $ThisLine =~ s/^….\/..\/.. ..:..:.. //;
print $ThisLine ;
}
}
# vi: shiftwidth=3 syntax=perl tabstop=3 et
Once you save the file, we now need to set the proper permissions:
chmod 755 applylongdate
Configure The Log Files
Next we need to tell Logwatch where the OSSEC log files are located. Anytime you add new log files or create new services to monitor in Logwatch, you should place your changes under the /etc/logwatch directory. We are going to create two configuration files. The first will handle OSSEC messages, and the second will handle OSSEC alerts and active response changes.
Let’s start by creating the configuration file for the main OSSEC log file:
cd /etc/logwatch/conf/logfiles
vi ossec.conf
The contents of the file should be as followed:
LogFile = /var/ossec/logs/ossec.log
*ApplyLongDate =
You can now save and exit the file. Next, we will create the config file for the remaining log files:
vi ossec-alert.conf
The contents of this file should be as followed:
LogFile = /var/ossec/logs/active-responses.log
LogFile = /var/ossec/logs/alerts/alerts.log
LogFile = /var/ossec/logs/firewall/firewall.log
Once complete, save and exit. The default permissions should be acceptable for our setup.
Configuring OSSEC Services
Next, we need to define the OSSEC services and identify what we want to use as a title when reports are generated. Here’s how to create the first file:
cd /etc/logwatch/conf/services
vi ossec.conf
The contents of this file is pretty simple:
Title = “OSSEC Messages”
LogFile = ossec
Once complete you can save and exit. We need to create one more file in this directory:
vi ossec-alert.conf
The contents of this file should be:
Title = “OSSEC Alerts”
LogFile = ossec-alert
Once complete, save and exit as usual.
Parsing The Entries
Next, we need to tell Logwatch how to format the log entries within the report. We will need to create a customization script for each set of services. We are going to start by using a Logwatch supplied test script; just to make sure everything works.
Start by moving into the appropriate directory:
cd /etc/logwatch/scripts/services
Use your favorite editor to create your first script:
vi ossec
The contents of the script should be as followed:
#!/bin/bash
# This is as nice script that will show you the lines you will
# be processing and reporting on. It will first display the
# standard environment variables and then it takes STDIN and
# dump it right back out to STDOUT.
# These are the standard environment variables. You can define
# more in your service config file (see above).
echo “Date Range: $LOGWATCH_DATE_RANGE”
echo “Detail Level: $LOGWATCH_DETAIL_LEVEL”
echo “Temp Dir: $LOGWATCH_TEMP_DIR”
echo “Debug Level: $LOGWATCH_DEBUG”
# Now take STDIN and dump it to STDOUT
cat
Now create your second script:
vi ossec-alert
and include the exact same contents:
#!/bin/bash
# This is as nice script that will show you the lines you will
# be processing and reporting on. It will first display the
# standard environment variables and then it takes STDIN and
# dump it right back out to STDOUT.
# These are the standard environment variables. You can define
# more in your service config file (see above).
echo “Date Range: $LOGWATCH_DATE_RANGE”
echo “Detail Level: $LOGWATCH_DETAIL_LEVEL”
echo “Temp Dir: $LOGWATCH_TEMP_DIR”
echo “Debug Level: $LOGWATCH_DEBUG”
# Now take STDIN and dump it to STDOUT
cat
Finally, we need to set the appropriate permissions:
chmod 755 ossec*
Testing The Setup
The easiest way to test our new setup is to run the command:
logwatch | less
If you only want to see your changes, you can run a report on each service, one at a time:
logwatch –service ossec | less
logwatch –service ossec-alert | less
Further Customization
Once you get all of the above working, you can focus on getting Logwatch to filter and summarize the log entries. Logwatch is pretty flexible, and you can customize the output any way you want. One of the nice things about the above test script above is that it shows you exactly what you have to work with. So with a little regular expression magic you can summarize the entries as you deem appropriate. For some ideas, check out the files located under:
/usr/share/logwatch/scripts/services
These are the default summary scripts included with Logwatch. Specifically, have a look at the files “pam” and “sshd”. They are great examples of both a simple and a complex set of summary filters.
As you develop your scripts, pay close attention to the $LOGWATCH_DETAIL_LEVEL” variable. This will permit you to customize the output level of the report depending on how much verbosity the user is looking for. For example, while you are still in the above services directory, run the following command:
less sshd
When the first page of the file’s contents is displayed, type in:
/Detail<Enter Key>
The backslash lets us search the file for a particular text string. In this case we are searching for the word “Detail”. Once you press Enter the search will skip through the file till it finds the first instance of the text string. It will also highlight the search string. In the first match you will notice that the author assigned the variable “$Detail” to be the same as the variable $LOGWATCH_DETAIL_LEVEL”. This is to save them some typing.
Now press the backslash key again followed by the Enter key. This will jump through the file to the next instance of “Detail”. You should see:
if ($Detail >= 20) {
$Users{$User}{$Host}{$Method}++;
} else {
if ( $Host !~ /$IgnoreHost/ ) {
$Users{$User}{$Host}{“(all)”}++;
Note the author provides more information if the detail level is set to 20 (half way between low and medium) or higher. Keep jumping through the file and you will see other examples where the author leveraged this technique.
Now page down to the end of the file and you should see this statement:
if (keys %OtherList) {
print “\n**Unmatched Entries**\n”;
print “$_ : $OtherList{$_} time(s)\n” foreach keys %OtherList;
}
This section is extremely important, as it is a final catchall. Think of a firewall policy for a moment. Most of us create a final rule that says, “If I did not specifically permit a traffic pattern through, deny it”. In other words, if something unexpected happens, this is how I want you to handle it.
The above statement serves the same purpose when parsing the logfile. All of the previous “if” statements attempt to match a specific text string in the log entry in order to format it properly. This entry says “If you have not matched and printed a specific log entry yet, print it out in a section titled “**Unmatched Entries**”. This step is extremely important because without it we will never see unexpected entries. It is the unexpected entries that are probably the most important and most interesting.
Exec Summary
Both OSSEC and Logwatch are excellent complimentary tools. OSSEC excels at warning you when a known attack pattern takes place. Logwatch is a terrific tool for summarizing a time chunk of logs so humans can actually make sense of what is going on. By combining the two tools you can create a far more robust defense in-depth posture. The whole becomes greater than the sum of the parts.
