Anyone that has trained with me can tell you I’m real big on being able to read your own detects. While we have plenty of security devices that try to accurately describe what they think they see on the wire, they are programmed by humans and humans make mistakes. Try and automate the process and the mistakes become compounded. Even Cisco has backed off a bit on their grandiose claims of what a self defending network is actually capable of. Nothing replaces having a skilled analyst reviewing the findings.

Good help is hard to find

Of course the keyword in that last comment is skilled. I’ve dealt with plenty of senior security folks that have never seen a decode of an IP packet, let alone can tell you what a legit IP session should look like. One of the problems is when we need training we usually turn to the vendors. Vendors tend to focus on their pretty GUI, not what’s going on behind the scenes.

In a previous life I owned an ISP and had some very entertaining abuse reports submitted. One of my personal favorites was an admin reporting that one of my systems was “sending hostile ICMP packets” to one of his systems. When I reviewed my logs, I noted that one of my routers was in fact sending him ICMP host unreachable messages. This would happen every time his host probed the RPC port of an IP address that was not in use. I wrote back and explained that if his system would simply stop probing for non-existent systems, my router would stop telling him the host is off-line.

Another admin (at a rather large, well known company I might add) informed me that one of my systems was attacking him with Code Red via e-mail. If you remember Code Red, it only attacked IIS Web servers via HTTP. The “attacks” in question were users subscribed to a mailing list. Folks were talking about how to write good intrusion signatures to properly catch Code Red. If that was not ironic enough, the payload of the decode he sent me as evidence explained that the attacks were only HTTP based. If that twist is still not enough to make you chuckle, he later admitted that he was the one of the people subscribed to that list.

The more things change the more they stay the same

My hosting provider Host Monster (a subsidiary of Blue Host) has put a filter in place blocking all access to the SANS Institute mail server (SysAdmin, Audit, Network, Security Institute; provides computer security training), The Internet Storm Center (daily diary of Internet security threats) and DShield (an early warning system for Internet threats). I contacted support and they confirmed they are filtering these sites. I was unable to find out why beyond “due to suspicious activity”.

I know the folks that maintain the SANS and Dshield servers. They are hard core security folks with a serious clue. When I first signed up with my hosting provider I was impressed with the knowledge level of their support personnel. Lately however, I’ve found them to be lacking in even the basics. While I’m left to guess as to what actually caused the ban, I’m inclined to think that someone at Host Monster (or possibly Blue Host) saw an alert but didn’t have the skills to figure out its a false positive.

Communication is a two way street

Blue Host claims 1.5 million hosted sites through all of their holdings. So they now have 1.5 million clients that can’t:

• Receive real time blocking alerts of malicious IP’s
• Receive assessments on current Internet threats
• Receive info on what’s going on in the security industry

So while attempting to protect themselves is a positive thing, the implementation has had a negative effect on the security of their clients.

How to verify a detect

So let’s pull something positive out of all of this and identify the proper procedure for verifying a security alert. We first need to start with good gear. Do not even consider an intrusion detection or prevention system that does not include:

• Access to the signature language
• Full decode of suspect packets

Without these features you are shooting in the dark.

Step 1: Understand the attack

When an alert gets triggered, make sure you understand the attack mechanism. What ports or services does it go after? Are there any known signatures? If you Google the attack’s name followed by the key words “false positive” and “spoofed”, does anything come up?

Step 2: Understand your intrusion system

No security product is perfect. They all have weaknesses or limitations. Does your intrusion system maintain state? If so, is it all the time or just some of the time? Does it properly validate CRC fields? How does it deal with fragmented traffic? Is it known to generate false positives? If so, are the false positives limited to only certain signatures or protocols, or is it all of the time?

Step 3: Sanity check the alert

Sometimes false positives can be weeded out from the limited amount of info presented in an alert. For example does the alert claim to have detected an HTTP attack coming from TCP/80 instead of going to it? If so, there is an obvious problem with the signature generating the alert.

Step 4: Check the signature

Some signatures are written very specifically so that there is little chance of a false positive. Some are more general however so its possible to have false positive fall out. Review the signature that generated the alert and make a judgement call. Does the signature check 3-4 different conditions or ten or more? Obviously the more parameters we are checking, the less likely we are to get a false positive.

Step 5: Check the decode

If you understand the attack pattern, you should already have an expectation of what will be in the attack decode. Does the packet match your expectations? I’ve seen plenty of false positives generated by people reading info on a Web site describing an HTTP based attack. These are easy to distinguish due to the extra HTML, proper agent and referrer fields, etc. In short, if the packet does not match a known decode of the real attack, figure out why.

Step 6: Research the source

I always take the time to make sure I understand who is sitting behind the source IP address. Sometimes this can go a long way towards identifying whether I can trust the alert. I’m reminded of a friend that banned a number of IP addresses his intrusion system had identified as hostile. Shortly after he started noticing that parts of the Internet were no longer reachable. Turns out someone spoofed a series of attacks from the IP addresses of the root name servers. Had he taken the time to look up the IP addresses first, he most certainly would not have blocked them.

Exec Summary

Blocking known to be hostile IP addresses can certainly be beneficial to security, but it must be implemented with caution. At the core of any network security system must be a knowledgeable security expert with good common sense. If that component is missing, the whole structure can fall apart like a house of cards.

Update: Since posting this I’ve found that Host Monster (Blue Host) is blocking access to one or more Cisco servers as well. Guess the list continues…