In my last post we installed Logwatch as well as OSSEC. It is now time to get Logwatch and OSSEC playing together in the same sandbox. In this post I’ll discuss how to get Logwatch to summarize the information being generated by OSSEC.

Deployment Options

We have two paths we can follow to set this up:

1. Have Logwatch parse the OSSEC logs directly.
2. Have OSSEC send its alerts to a Syslog type server, then run Logwatch on the Syslog server.

The benefit to option #1 is that we only need one system. Logwatch will be run on the system hosting the OSSEC server. The problem we are going to run into however involves the OSSEC alert file. Log entries do not get normalized. This means the format can change from entry to entry, and may even spread over multiple lines. It is going to be a real nightmare to create a Logwatch script that will filter and summarize the alert information.

If we go with option #2, we will require another box to act as our centralized logging server. In order to have the OSSEC server accept log entries from non-agent systems, it has to listen on UDP/514. This is the same port used by a centralized logging server, and you can’t have two applications share the same port (except with Windows, but socket access is extremely messy). On the plus side, the alert entries will get normalized when they are transmitted to the Syslog server so creating a Logwatch summary script will be far easier. Further, Logwatch already knows about the standard Syslog files, so we will have less customization work to do.

Finally, I mentioned in an earlier post that OSSEC is not designed to be a SIM. This is because it does not record everything, just the events that generate an alert. So we are probably going to want a centralized server anyway, and it makes sense to have it store the information being generated by OSSEC.

So if it sounds like I am steering you towards option #2, you are absolutely correct. With that said, I’m actually going to cover option #1 as it is a far more complex setup.

Dealing With Date/Time Stamps

Have a look at the main OSSEC logfile and you should see similar to the following:

[root@fubar logs]# tail -3 /var/ossec/logs/ossec.log

2010/02/18 12:32:05 ossec-rootcheck: INFO: Ending rootcheck scan.

2010/02/18 14:27:06 ossec-syscheckd: INFO: Starting syscheck scan.

2010/02/18 14:39:21 ossec-syscheckd: INFO: Ending syscheck scan.

Note the way the date/time stamp is formatted. This is different than most applications, so the first thing we will need to do is tell Logwatch how to deal with this format. We will need to create a script that we can call when needed that understands the format shown above.

Start by moving into the shared scripts directory:

cd /usr/share/logwatch/scripts/shared

Using your favorite editor, create a file named “applylongdate”:

vi applylongdate

Here is what you need inside of that file. Feel free to copy/paste from this page:

use Logwatch ‘:dates’;

my $Debug = $ENV{‘LOGWATCH_DEBUG’} || 0;

$SearchDate = TimeFilter(‘%Y/%m/%d %H:%M:%S’);

if ( $Debug > 5 ) {

print STDERR “DEBUG: Inside ApplyLongDate…\n”;

print STDERR “DEBUG: Looking For: ” . $SearchDate . “\n”;

}

while (defined($ThisLine = <STDIN>)) {

if ($ThisLine =~ m/^$SearchDate /o) {

print $ThisLine =~ s/^….\/..\/.. ..:..:.. //;

print $ThisLine ;

}

}

# vi: shiftwidth=3 syntax=perl tabstop=3 et

Once you save the file, we now need to set the proper permissions:

chmod 755 applylongdate

Configure The Log Files

Next we need to tell Logwatch where the OSSEC log files are located. Anytime you add new log files or create new services to monitor in Logwatch, you should place your changes under the /etc/logwatch directory. We are going to create two configuration files. The first will handle OSSEC messages, and the second will handle OSSEC alerts and active response changes.

Let’s start by creating the configuration file for the main OSSEC log file:

cd /etc/logwatch/conf/logfiles

vi ossec.conf

The contents of the file should be as followed:

LogFile = /var/ossec/logs/ossec.log

*ApplyLongDate =

You can now save and exit the file. Next, we will create the config file for the remaining log files:

vi ossec-alert.conf

The contents of this file should be as followed:

LogFile = /var/ossec/logs/active-responses.log

LogFile = /var/ossec/logs/alerts/alerts.log

LogFile = /var/ossec/logs/firewall/firewall.log

Once complete, save and exit. The default permissions should be acceptable for our setup.

Configuring OSSEC Services

Next, we need to define the OSSEC services and identify what we want to use as a title when reports are generated. Here’s how to create the first file:

cd /etc/logwatch/conf/services

vi ossec.conf

The contents of this file is pretty simple:

Title = “OSSEC Messages”

LogFile = ossec

Once complete you can save and exit. We need to create one more file in this directory:

vi ossec-alert.conf

The contents of this file should be:

Title = “OSSEC Alerts”

LogFile = ossec-alert

Once complete, save and exit as usual.

Parsing The Entries

Next, we need to tell Logwatch how to format the log entries within the report. We will need to create a customization script for each set of services. We are going to start by using a Logwatch supplied test script; just to make sure everything works.

Start by moving into the appropriate directory:

cd /etc/logwatch/scripts/services

Use your favorite editor to create your first script:

vi ossec

The contents of the script should be as followed:

#!/bin/bash

# This is as nice script that will show you the lines you will

# be processing and reporting on.  It will first display the

# standard environment variables and then it takes STDIN and

# dump it right back out to STDOUT.

# These are the standard environment variables.  You can define

# more in your service config file (see above).

echo “Date Range: $LOGWATCH_DATE_RANGE”

echo “Detail Level: $LOGWATCH_DETAIL_LEVEL”

echo “Temp Dir: $LOGWATCH_TEMP_DIR”

echo “Debug Level: $LOGWATCH_DEBUG”

# Now take STDIN and dump it to STDOUT

cat

Now create your second script:

vi ossec-alert

and include the exact same contents:

#!/bin/bash

# This is as nice script that will show you the lines you will

# be processing and reporting on.  It will first display the

# standard environment variables and then it takes STDIN and

# dump it right back out to STDOUT.

# These are the standard environment variables.  You can define

# more in your service config file (see above).

echo “Date Range: $LOGWATCH_DATE_RANGE”

echo “Detail Level: $LOGWATCH_DETAIL_LEVEL”

echo “Temp Dir: $LOGWATCH_TEMP_DIR”

echo “Debug Level: $LOGWATCH_DEBUG”

# Now take STDIN and dump it to STDOUT

cat

Finally, we need to set the appropriate permissions:

chmod 755 ossec*

Testing The Setup

The easiest way to test our new setup is to run the command:

logwatch | less

If you only want to see your changes, you can run a report on each service, one at a time:

logwatch –service ossec | less

logwatch –service ossec-alert | less

Further Customization

Once you get all of the above working, you can focus on getting Logwatch to filter and summarize the log entries. Logwatch is pretty flexible, and you can customize the output any way you want. One of the nice things about the above test script above is that it shows you exactly what you have to work with. So with a little regular expression magic you can summarize the entries as you deem appropriate. For some ideas, check out the files located under:

/usr/share/logwatch/scripts/services

These are the default summary scripts included with Logwatch. Specifically, have a look at the files “pam” and “sshd”. They are great examples of both a simple and a complex set of summary filters.

As you develop your scripts, pay close attention to the $LOGWATCH_DETAIL_LEVEL” variable. This will permit you to customize the output level of the report depending on how much verbosity the user is looking for. For example, while you are still in the above services directory, run the following command:

less sshd

When the first page of the file’s contents is displayed, type in:

/Detail<Enter Key>

The backslash lets us search the file for a particular text string. In this case we are searching for the word “Detail”. Once you press Enter the search will skip through the file till it finds the first instance of the text string. It will also highlight the search string. In the first match you will notice that the author assigned the variable “$Detail” to be the same as the variable $LOGWATCH_DETAIL_LEVEL”. This is to save them some typing.

Now press the backslash key again followed by the Enter key. This will jump through the file to the next instance of “Detail”. You should see:

if ($Detail >= 20) {

$Users{$User}{$Host}{$Method}++;

} else {

if ( $Host !~ /$IgnoreHost/ ) {

$Users{$User}{$Host}{“(all)”}++;

Note the author provides more information if the detail level is set to 20 (half way between low and medium) or higher. Keep jumping through the file and you will see other examples where the author leveraged this technique.

Now page down to the end of the file and you should see this statement:

if (keys %OtherList) {

print “\n**Unmatched Entries**\n”;

print “$_ : $OtherList{$_} time(s)\n” foreach keys %OtherList;

}

This section is extremely important, as it is a final catchall. Think of a firewall policy for a moment. Most of us create a final rule that says, “If I did not specifically permit a traffic pattern through, deny it”. In other words, if something unexpected happens, this is how I want you to handle it.

The above statement serves the same purpose when parsing the logfile. All of the previous “if” statements attempt to match a specific text string in the log entry in order to format it properly. This entry says “If you have not matched and printed a specific log entry yet, print it out in a section titled “**Unmatched Entries**”. This step is extremely important because without it we will never see unexpected entries. It is the unexpected entries that are probably the most important and most interesting.

Exec Summary

Both OSSEC and Logwatch are excellent complimentary tools. OSSEC excels at warning you when a known attack pattern takes place. Logwatch is a terrific tool for summarizing a time chunk of logs so humans can actually make sense of what is going on. By combining the two tools you can create a far more robust defense in-depth posture. The whole becomes greater than the sum of the parts.

In my last two posts I discussed Logwatch and OSSEC, as well as how they can be leverage to augment your security posture. In this installment I’ll discuss how to install both of these tools.

Installing Logwatch

Logwatch is pretty easy to install. In fact, it is installed by default on many Linux distributions so you may already have a copy on your system. To check, logon as root and try running Logwatch with the “-v” switch. If you see:

[root@fubar logwatch]# logwatch -v

Logwatch 7.3.6 (released 05/19/07)

Logwatch is installed and you have a copy of the latest version. If you do not have the latest version, you can grab it from the Logwatch download page.

There are three flavors of Logwatch which can be downloaded; Binaries in RPM format, source in RPM format, or source in a Tar ball. If your system supports RPM package management, the binary RPM is your best choice. It is simple to install and RPM will automatically update the software when new versions are available.

Installing Logwatch From RPM

To install the binary version of the RPM, simply logon as root and navigate to the directory where you downloaded the RPM file. Now execute the command:

rpm -U logwatch-7.3.6-1.noarch.rpm

Don’t forget you can use the tab key to auto-complete the file name rather than having to type in the whole thing.

Installing Logwatch From Source

Installing from source is a bit more time consuming. Remember that in order to install source code you must already have a compiler (like gcc) installed on your system. Logon as root and navigate to the directory where you downloaded the Tar ball. To extract the archive, execute the command:

tar xvzf logwatch-7.3.6.tar.gz

You will see a directory structure below your current location get created and lots of files being copied in. We now need to move into the top most directory that was created:

cd logwatch-7.3.6

In order for Logwatch to run, there are a bunch of directories that need to be created on your system. These are documented in the README in the current directory. Luckily, Logwatch includes an install script that can do all the work for you. Unfortunately, the script has the wrong permissions set so it will not run by default. This is pretty easy to fix however with the chmod command:

chmod 500 install_logwatch.sh

Now we can run the script to setup our system:

./install_logwatch.sh

Don’t forget the period at the beginning of the line.

Testing Logwatch

To test your Logwatch setup, execute the command:

logwatch | less

You will see your terminal screen go blank, but that is normal. You will eventually see a Logwatch report get printed to the screen that you can navigate through using the “Page Up” and “Page Down” keys. How log it takes for the report to show up on the screen will depend on how much log information needs to get parsed. It could take a few seconds or a couple of minutes. Either way, it will give you a chance to familiarize yourself with the report format.

Installing OSSEC

As I mentioned in my last post, you have two installation options with OSSEC, local or client/server. In this post I’m going to focus on the client/server setup, as it is a bit more complex. If you are performing a local install, simply select the “local” option during the install process and skip the section on setting up a secure channel between the agent and the server.

Start With The Server

OSSEC uses Blowfish encryption to secure communication between the client and the server. Blowfish is symmetrical key based, so both sides must know what key value to use in order to communicate. The server is responsible for generating the symmetrical key, so we have to install the server software first. During the client install we will be prompted for a key value so obviously we will need to have that handy ahead of time.

Here’s a time saving tip. The key value is long and nearly impossible to remember. The easiest way to move the key value from the server system to the agent system is to use SSH. Create a secure connection to the OSSEC server, and extract the appropriate key (directions provided below). In a second terminal window, create an SSH session to the system where you will be installing the agent. When the client install prompts you for the key value, you can simply copy/paste between the two terminals.

Installing The OSSEC Server

The server software is available as a Tar ball, so you can grab a copy of the latest version from the OSSEC download page. You will then need to extract the contents of the Tar ball:

tar xvzf ossec-hids-2.3.tar.gz

Next, move into the directory structure you just created:

cd ossec-hids-2.3

OSSEC provides an install script to walk you through the process of installing the server. To start the script, type:

./install.sh

Don’t forget the period at the beginning of the command. You will now be prompted through a number of install options:

• Language – The default is English. Change as needed.
• Confirmation of installation – Press Enter once you have read the screen.
• Install type – Type in “server” without the quotes and press Enter.
• Install location – Accept the default.
• E-mail notification – Default is yes, select if you want e-mail alerts. If you select yes, you will be prompted for a valid e-mail address and mail server.
• Integrity check – Default is yes. Select if you want the local system periodically checked for intrusions.
• Root kit detection – Default is yes. Good option since we need to maintain a high level of integrity on this system.
• Active response – Default is yes. Select this option if you wish to be able to respond to events.
• Firewall drop – Permits the OSSEC server to defend it self if a direct attack is detected.
• White list – This will permit you to add IP addresses from which possible attacks will be ignored. Be careful with this option. If you will not have console access to the OSSEC server, it might be wise to identify one IP address that can always get in. Just ensure the source IP is a trustworthy system.
• Enable Syslog – Default is yes. Select this option if you wish to collect logs from system that cannot run an OSSEC agent (like firewalls, switches, routers, access points, etc.).
• Log files to be monitored – This screen identifies all of the local log files OSSEC will monitor. It is purely information, so all you can do is press Enter to move past it. If you notice one or more log files missing from the list, you can add them later to the ossec.conf file.

At this point OSSEC will access the local complier and install all needed files onto the system. Once complete, you can start the OSSEC server by executing the command:

/var/ossec/bin/ossec-control start

Defining OSSEC Agents

We are not done with the OSSEC server just yet. Next, we need to pre-define any systems that will be running the OSSEC agent (client) software. This is performed using the manage_agents command. First however, we need to do a bit of homework. Make a list of all of the systems that will be running the OSSEC agent software. For each system, you will need a descriptive name as well as that system’s IP address.

Now, execute the following from the command line:

/var/ossec/bin/manage_agents

This will produce the Agent Manager main menu. Press “A” followed by the Enter key to define your first system. Enter a descriptive name for the first system, followed by the system’s IP address. Don’t worry about the agent ID number. Simply accept the default and OSSEC will auto-assign the next available ID number. Once you confirm the information you entered, you will be returned to the Agent Manager main menu. Repeat the above process for each system that will be running an OSSEC agent.

Generating Keys

Once you have added in all of your systems, it is time to generate encryption keys. This step is also performed with the manage_agents utility. If you closed the tool after the last step, relaunch it now.

Press the “E” key followed by Enter to select the “Extract key for an agent” menu option. You will then be prompted for the ID number of the key you wish to extract. The descriptive names and IP addresses are listed with each ID number, so it should be trivial to identify which one you want. Start with the system you plan to install the agent software onto first.

OSSEC Agent Install On Linux

When installing the agent software on a Linux or UNIX client, you use the exact same Tar ball we used to install the server software. Run the same install script, but this time when you are prompted for the type of install you wish to perform, type in “agent” followed by the Enter key.

The client install has many of the same prompts as the server install. Use the info above to guide you through the process. The prompt that will vary however is that you will be asked to provide the IP address of the OSSEC server. Once complete, OSSEC will access the local complier and install all required files onto the system.

Next we need to import the Blowfish key from the OSSEC server. While still on the agent system, run the command:

/var/ossec/bin/manage_agents

When the Agent Manager menu appears, select “I” to import the Blowfish key.

When the next prompt appears, you need to manually enter the appropriate Blowfish key. Again, if you are running SSH to both systems at the same time, you can simply copy/paste between the two terminals. Make sure the key looks correct, press the Enter key, and then select “y” to confirm that the key looks correct. You will be returned to the Agent Manager menu. Select “q” in order to return to the command line.

Now we simply need to start the OSSEC agent. You can do so by executing the following command:

/var/ossec/bin/ossec-control start

You should see all of the OSSEC agent components start up, followed by a “Completed” message.

OSSEC Agent Install On Windows

OSSEC has a self-extracting executable that will permit you to install the agent software on a Windows system. Simply double click the executable to start the install process. You will be prompted to agree to the license as well as which components you wish to install. Simply follow the prompts till the OSSEC Agent Manager window appears.

The OSSEC Agent Manager window will prompt you for the IP address of the OSSEC server. It will also prompt you for the Blowfish key value to use, so extract the appropriate key on the server and enter the value in this field. Make sure you delete the prompt within this field before you paste in the Blowfish key. Otherwise communication with the server may fail.

Next, select “Manage” from the OSSEC Agent Manager menu, followed by “Start OSSEC”. You should now see the “Status:” indicator change to “Running…”.

Testing OSSEC

Once you have the server and agent software installed, started and the appropriate keys configured, it is now time to check our setup. Execute the following command on the OSSEC server:

cd /var/ossec/logs

And check out the ossec.log file:

less ossec.log

Check the log file for any errors. A common error is that OSSEC reports it cannot send e-mail. Make sure the mail server is running and that it is accepting connections. Once you are happy with the server setup, it is now time to check out the agents. Move down to the “alerts” directory:

cd alerts

And check out the alerts.log file:

less alerts.log

Specifically, you are looking for entries similar to the following:

2010 Feb 17 16:09:16 (desktop) 192.168.1.10->ossec

Rule: 501 (level 3) -> ‘New ossec agent connected.’

Src IP: (none)

User: (none)

ossec: Agent started: ‘test_system->192.168.1.10′.

You should see an entry for every system on which you installed the agent software.

More To Come

Whew! That’s more than enough for one post! In my next post I’ll get into leveraging Logwatch to parse all of the alert information being generated by OSSEC.

In my last post I described how Logwatch could be used to simplify the log review process. In this post we’ll look at OSSEC and what it brings to the table.

What Is OSSEC?

OSSEC, short for “Open Source SECurity”, is a host based intrusion detection system (HIDS). In other words, it is designed to detect attacks or policy violations if and when they occur. While it does not have the ability to protect against unknown or 0-Day attacks (that would be host based intrusion prevention), it does include a wide range of tools which can help you identify an intrusion when it occurs, as well as the extent of the damage that has been caused.

Supported Platforms

To take advantage of all of the features OSSEC has to offer, you have to run an agent on the system being protected. OSSEC agents can run on Windows, Mac OS X, Linux, and a wide range of UNIX systems. If you are just interested in the log analysis portion however, an even wider range of systems can be supported. This includes hardware from both Cisco and Juniper. A number of specific products are also supported like Checkpoint firewalls, Symantec Anti-Virus, Snort, Squid, and Arpwatch, just to name a few.

When you install OSSEC you have two configuration options, local or client/server. A local install is used when you need to run everything on a single system. The client/server installation lets you run a distributed environment protecting multiple systems at the same time. While most deployments are client/server based, if you want to give OSSEC a spin you can easily run everything on a single test system using a local install.

Log Analysis

OSSEC includes a Log-based Intrusion Detection System (LIDS). This has the ability to review log files in near real time, while scrutinizing them for known attack patterns. When a log is generated on a protected system, the agent takes care of securely transmitting the log (Blowfish encryption using a pre-shared secret) back to the server. The server then takes care of performing the analysis.

Most log analysis tools process their rules in a linear format. By that I mean if we have 500 rules, rule one is checked, then rule two, then rule three and so on till a match is found or we reach the end of the rule set.  OSSEC works a bit differently as it implements a hieratical structure to the rules. Log entries are first classified and then checked only against whichever rules are appropriate. The result is that rather than needing to process all 500 rules, most events will get checked against 10 rules or less. This dramatically reduces the amount of overhead required to process the rule set.

Integrity Checking

OSSEC includes a tool called Syscheck for performing file and directory integrity checking. If you are running a Windows agent, you can also include specific keys within the Windows registry to be monitored as well. File changes can be detected using both MD-5 and SHA-1 hash algorithms. The system is extremely customizable. You can include or exclude single files, or entire directory structures. You can even set a flag to detect new file creation.

The agent software is designed to use a minimal amount of CPU during the integrity check. While this means the check will take longer, it also helps to minimize the performance hit to the system. Hash information is transmitted back to the server. The server then takes care of performing the hash comparison to see if any of the system’s critical files have been changed. The server also stores a copy of the integrity check policy, so that if policy changes are made on the agent, they can be detected and reported as well.

Anomaly detection

OSSEC goes far beyond log checking to verify system integrity. Usage policies can be centrally managed from the server, and then pushed out to the appropriate agents. For example you could define a policy regarding which Windows applications are acceptable (Office, Firefox, etc.) and which ones are not (IM client, Skype, etc.). You can even define acceptable configuration options like verifying that NT hashes are being used for password stored but not LanMan hashes.

OSSEC includes a number of other goodies in order to help verify a system’s integrity. For example OSSEC has the ability to execute commands from the agent and monitor the output that gets generated. For example you could have the Linux agent execute the “df” command at regular intervals and generate an alert if disk usage exceeds 90%. A Windows example may be to have OSSEC generate an alert whenever file information is written to the alternate data streams area of NTFS.

Active Response

Finally, OSSEC includes the ability to respond when suspicious activity is detected. Responses can be generated from the server or the agent, which ever you specify. Responses can be as benign as generating an e-mail alert, to being as proactive as blocking a remote IP address for a limited amount of time. There are a number of included active response scripts you can draw on, or you can easily write your own.

Secure Architecture

The OSSEC authors have gone to great lengths to secure all of the components within the product. Tasks such as integrity checking are performed on the server, rather than the agent, so the trustworthiness of the hashes cannot be compromised during an attack. Processes are run with the lowest level of permissions possible, and different accounts are used to run each OSSEC component. This means that a compromise of a single application within OSSEC will not immediately lead to a compromise of the full package. Further, most components are run within a chrooted jail so their access to the system is even further restricted.

Final Words

While OSSEC is a powerful tool, it is important to remember that it is a HIDS and not a log management solution. OSSEC can review log entries looking for suspicious patterns, but it will only save alert information. So while OSSEC will not replace your Security Information Management (SIM) solution, it can most certainly augment it. You can easily setup OSSEC to forward all alerts it generates to a central logging server.

While OSSEC is open source software, Trend Micro is primarily developing it. If you need commercial support, you can purchase a support contract through them at a reasonable fee.

More To Come

In my next installment we’ll look at installing OSSEC and Logwatch. After that, we’ll move into integrating the two together.

I recently had a student ask me a question regarding the integration of Logwatch with OSSEC. I felt like this was a complex and yet cool enough idea that it warranted a series of posts to cover it in full. So over the next few days I’ll talk about each of these tools, how to integrate them together, as well as what additional security visibility can be gained once the process is complete.

What Is Logwatch?

Logwatch is an excellent open source tool for generating daily human readable log reports. Log entries tend to fall into one of three categories:

• Stuff you know is evil
• Stuff you know is normal and can be safely ignored
• Everything else

It is that “everything else” category where Logwatch really shines. For the stuff we know is evil, we will setup some form of alerting system. For example we may write an alert signature that warns the security analyst when an account is being brute forced. But what about the attacks we don’t know about or are not sure what they look like? This would be a clear example of that “everything else” category. The traffic is not normal, but we have not seen it before to have a signature waiting to generate an alert. Since we will be unable to catch the attack in real time, we will need to catch it during a daily log review.

Of course the problem with doing daily log reviews is that it is tedious and time consuming. I mean let’s be honest, who really wants to spend their day reviewing one million plus log entries? Even if you did, are you sure you would actually catch the out of the ordinary traffic?

How It Works

What Logwatch does very well is permit you to reorganize your data into a format that is easier for humans to follow. Its real strength is that it permits you to move the stuff you understand out of the way (normal or evil), so that the unexpected log entries stand out like a sore thumb. In other words, Logwatch lets you summarize your log entries so the unusual stuff is easier to spot.

What I really love about Logwatch is that you don’t lose anything. Many log review tools will only show you the stuff that has been pre-defined as being evil. The problem they all share is that when something evil but unexpected happens, it flies right under the wire. Because Logwatch lets you see everything, you no longer miss the unexpected.

Logwatch In Action

Lets discuss how Logwatch works using the SSH server service as an example. The scripts to deal with SSH have already been defined within Logwatch, so you do not need to do any tweaking to receive the features we are going to discuss.

When reviewing a log file, the first thing Logwatch does is reorganize log entries based on their message types. For example all Successful SSH logons are grouped together, as well as too many logon failures, refused connections, locked accounts, accounts without a proper shell, protocol mis-matches, etc. etc. etc. Once all the SSH messages are grouped by their type, the data is then summarized to reduce the amount of information being reported.

For example, the default is to summarize failed logon attempts by account and source IP. So a typical failed logon report section might look like this:

Failed logins from these:

bsmith/password from 1.2.3.4: 637 time(s)

jsmith/password from 1.2.3.5: 2 time(s)

So rather than having to review 639 log entries reporting a bad logon attempt, we have all the pertinent information summarized into three lines (if you include the title). Continue this process for all of the other SSH messages, and we have dramatically reduced the amount of time required to review our logs.

But what if something happens that Logwatch is not pre-programmed to recognize? When an unexpected log entry is found, Logwatch adds a section to the end of the service report called “Unmatched Entries”. So if we see this title in the SSH server section, we know some event has occurred that is either abnormal or unexpected for the SSH service. This could very well be some form of attack that we are not aware is making the rounds.

By focusing in on the unmatched entries section, we can quickly identify unexpected activity. As I stated earlier, this is really the main goal of doing daily log reviews. To find the stuff we don’t expect which will sneak past our alerting system. Logwatch makes this process as quick and as painless as possible.

Feature Summary

In the above example I talked about doing daily log reviews, but to be honest Logwatch is highly customizable. You can specify any range you wish to use down to an interval of one second. For example let’s say I’m investigating an intrusion rather than performing a daily log review. I could specify a range such as “2010/02/14 17:05:00 for that hour” to focus right in on the information that interest me. I can also focus in on one specific log file or service.

The detail level of the report is also customizable. Typically when you deal with security you get in the habit of always wanting the highest detail level of reporting. To be honest, with Logwatch a high level of detail is probably more than you will ever need. Personally I typically stick with “med” for medium and that works out nicely. You can also specify the reporting level as “low” or “high” or use a numeric range of 0-10 for a higher level of granularity (low =0, med=5, high=10).

Logwatch can be run automatically or as a manual process. Typically you will want to set it up to run automatically each day and summarize one day’s worth of log entries. If you ever need to expand or focus the report, you can always run Logwatch from the command line while specifying exactly what you want to see. You can then use the “–save” option to specify a report name and directory location for storage.

More To Come

The above should give you a good idea as to the features Logwatch can bring to the table. In the next post I’ll discuss OSSEC in the same level of detail. After that, I’ll get into how to install each tool as well as how to integrate them together.