I recently had a student ask me a question regarding the integration of Logwatch with OSSEC. I felt like this was a complex and yet cool enough idea that it warranted a series of posts to cover it in full. So over the next few days I’ll talk about each of these tools, how to integrate them together, as well as what additional security visibility can be gained once the process is complete.
What Is Logwatch?
Logwatch is an excellent open source tool for generating daily human readable log reports. Log entries tend to fall into one of three categories:
- Stuff you know is evil
- Stuff you know is normal and can be safely ignored
- Everything else
It is that “everything else” category where Logwatch really shines. For the stuff we know is evil, we will setup some form of alerting system. For example we may write an alert signature that warns the security analyst when an account is being brute forced. But what about the attacks we don’t know about or are not sure what they look like? This would be a clear example of that “everything else” category. The traffic is not normal, but we have not seen it before to have a signature waiting to generate an alert. Since we will be unable to catch the attack in real time, we will need to catch it during a daily log review.
Of course the problem with doing daily log reviews is that it is tedious and time consuming. I mean let’s be honest, who really wants to spend their day reviewing one million plus log entries? Even if you did, are you sure you would actually catch the out of the ordinary traffic?
How It Works
What Logwatch does very well is permit you to reorganize your data into a format that is easier for humans to follow. Its real strength is that it permits you to move the stuff you understand out of the way (normal or evil), so that the unexpected log entries stand out like a sore thumb. In other words, Logwatch lets you summarize your log entries so the unusual stuff is easier to spot.
What I really love about Logwatch is that you don’t lose anything. Many log review tools will only show you the stuff that has been pre-defined as being evil. The problem they all share is that when something evil but unexpected happens, it flies right under the wire. Because Logwatch lets you see everything, you no longer miss the unexpected.
Logwatch In Action
Lets discuss how Logwatch works using the SSH server service as an example. The scripts to deal with SSH have already been defined within Logwatch, so you do not need to do any tweaking to receive the features we are going to discuss.
When reviewing a log file, the first thing Logwatch does is reorganize log entries based on their message types. For example all Successful SSH logons are grouped together, as well as too many logon failures, refused connections, locked accounts, accounts without a proper shell, protocol mis-matches, etc. etc. etc. Once all the SSH messages are grouped by their type, the data is then summarized to reduce the amount of information being reported.
For example, the default is to summarize failed logon attempts by account and source IP. So a typical failed logon report section might look like this:
Failed logins from these:
bsmith/password from 18.104.22.168: 637 time(s)
jsmith/password from 22.214.171.124: 2 time(s)
So rather than having to review 639 log entries reporting a bad logon attempt, we have all the pertinent information summarized into three lines (if you include the title). Continue this process for all of the other SSH messages, and we have dramatically reduced the amount of time required to review our logs.
But what if something happens that Logwatch is not pre-programmed to recognize? When an unexpected log entry is found, Logwatch adds a section to the end of the service report called “Unmatched Entries”. So if we see this title in the SSH server section, we know some event has occurred that is either abnormal or unexpected for the SSH service. This could very well be some form of attack that we are not aware is making the rounds.
By focusing in on the unmatched entries section, we can quickly identify unexpected activity. As I stated earlier, this is really the main goal of doing daily log reviews. To find the stuff we don’t expect which will sneak past our alerting system. Logwatch makes this process as quick and as painless as possible.
In the above example I talked about doing daily log reviews, but to be honest Logwatch is highly customizable. You can specify any range you wish to use down to an interval of one second. For example let’s say I’m investigating an intrusion rather than performing a daily log review. I could specify a range such as “2010/02/14 17:05:00 for that hour” to focus right in on the information that interest me. I can also focus in on one specific log file or service.
The detail level of the report is also customizable. Typically when you deal with security you get in the habit of always wanting the highest detail level of reporting. To be honest, with Logwatch a high level of detail is probably more than you will ever need. Personally I typically stick with “med” for medium and that works out nicely. You can also specify the reporting level as “low” or “high” or use a numeric range of 0-10 for a higher level of granularity (low =0, med=5, high=10).
Logwatch can be run automatically or as a manual process. Typically you will want to set it up to run automatically each day and summarize one day’s worth of log entries. If you ever need to expand or focus the report, you can always run Logwatch from the command line while specifying exactly what you want to see. You can then use the “–save” option to specify a report name and directory location for storage.
More To Come
The above should give you a good idea as to the features Logwatch can bring to the table. In the next post I’ll discuss OSSEC in the same level of detail. After that, I’ll get into how to install each tool as well as how to integrate them together.