In the last post I started counting down the five greatest threats to perimeter security. In this post I’ll complete the list.

Firewall Threat #3: Outbound HTTP

The popularity of HTTP (TCP/80) has become both a blessing and a tragedy. Certainly the Internet would not be as popular as it is today without the World Wide Web. While HTTP has lead to the greatest exchange of information in mankind’s history, our implementation of the service has caused it to become one of our greatest security problems on the Internet.

Why is it a threat?

The first issue is that TCP/80 access has become so commonplace; many firewall administrators have chosen to ignore it. An overwhelming majority of the networks I have audited permit outbound TCP/80 access but then never log its use. When I ask why the permitted traffic pattern is not being logged, the standard answer I receive is “it makes my firewall logs too big”. Hummm, didn’t realize the perimeter security mantra was “no fatties”.

Permitted traffic is inherently a higher risk than denied traffic because it facilitates the exchange of information. That passing traffic could be a zombie calling home or an internal system leaking sensitive data. If we do not log the use of a permitted protocol, we are completely blind to its abuse. The problem “how do I process large log files?” is much easier to solve than “how do I spot evil traffic when I’m not bothering to look for it?”.

Since HTTP has become a “turn it on and forget it” service, vendors and attackers alike have started running everything through this port. The brainchild at Microsoft who thought tunneling RPC through DCOM though HTTP was a good idea obviously had zero concern for how we would actually secure the implementation. While IRC used to be the protocol of choice for call home Malware, it is now HTTP because attackers can usually count on that port being wide open and unlogged.

How to prevent it

All permitted traffic patterns need to be logged. This includes outbound HTTP traffic traveling from the internal network to the Internet. In a later post I’ll tackle the problem of processing firewall log files so it is relatively easy to pull out the interesting bits.

Firewall threat #2: Banner grabbing

Most Internet based servers will happily identify themselves to connecting clients. For example whenever you connect to this hosted server, your browser sees:

Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8k DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635

While this information can be useful for troubleshooting, it can also be extremely useful for someone wanting to attack the system.

Why is it a threat?

Consider the following analogy: You are playing Five Card Draw against a number of opponents. Your opponent’s cards are well hidden in their hands, while your cards are laid out on the table for all to see. What are your chances of walking away the big winner at the end of the night?

Displaying version banners to connecting clients puts you in a similar position. If an attacker can see what software you are running along with the specific version, they can immediately determine if you are vulnerable to any of the attacks in their arsenal. So by displaying a software banner you’ve effectively helped the attacker get it right on the first try.

Without the benefit of the banner, the attacker would be forced to try each of their attacks in order to see if they will work. If we are vulnerable, we’re still going to get whacked. If we’re not, we’ve just forced the attacker to start generating log entries that will clue us in that the source IP is hostile. In other words, we’ve called their bluff so we now get to see their losing cards. This gives us an audit history and time to respond accordingly.

How to prevent it

Change the banners on any Internet facing services. This includes Web, FTP, name servers, mail servers, or any other service that can be reached from the Internet. Do not forget to restart the service after you make the change.

How easy or hard this is depends on the vendor. For example to fix this problem with the Apache Web server we would simply edit the “httpd.conf” file and change the “ServerTokens” parameter to be “Prod”. With IIS however, we do not have this type of flexibility. Microsoft does not let you change the banner to help insure they can properly identify their current market share. Your only real option is to put a reverse proxy in front of the Web server and leverage the proxy to scrub the banner.

Can changing the banner cause problems?

Most vulnerability scanners are primarily banner grabbing devices. For example when you run a vulnerability scanner against your mail server, it does not try every attack pattern its been programmed to test. Rather, it will grab the server’s banner and check it against a built in database. If the reported software version has known vulnerabilities, they get printed to a report. If you have ever run a vulnerability scan which claims to check for thousands of known attacks, but your IDS barely notices the scan, this is why.

Now with that said, not all checks are banner based. For example reading the banner will not tell the vulnerability scanner whether your mail server can be used as a spam relay. The scanner has to specifically test for that condition. So some exploits do need to be tested directly. Simply reading the banner however can satisfy a majority of the verification testing.

So, changing the banners not only makes it more difficult for attackers to assess your vulnerabilities, but it makes it more difficult for you to do so as well. You may be forced to drop to the command line to verify the version of software you are running. Luckily most software supports a “-v” or “-V” option which will print out it’s version information. Sometimes a different switch value is used so we will need to do a bit of research in the application’s help files. For example to get version information for Sendmail we would type:

[root@fubar ~]# sendmail -d0.1

Version 8.14.3

Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX

MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6

NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF SOCKETMAP STARTTLS

TCPWRAPPERS USERDB USE_LDAP_INIT

Firewall threat #1: Non-signature Malware

I’ve written extensively on the problems with detecting Malware. Feel free to use the search option on this site to pull up earlier posts for more info.

The bottom line is we try very hard to solve this problem at the perimeter by leveraging Unified Threat Management (UTM), firewall plug-ins, anti-virus proxies, etc. These solutions will never be 100% effective. In fact, their effectiveness has been declining sharply over the last few years. If you truly want to get a handle on modern day Malware threats you have to look at an application control solution.

Exec Summary

So the top five firewall threats are:

1. Non-signature Malware
2. Leaking banner info
3. Outbound HTTP
4. Outbound SSH
5. Commercial VPN services

Note that the last four of the five are outbound traffic patterns. While we tend to focus heavily on what is trying to get in to our network, we also tend to blindly trust the traffic leaving it. Its this misplaced confidence that has lead to each of these items making it to our list.

I spend a bit of time in the field consulting. Over the last few years I’ve noticed a trend with new clients whereby I can usually (about 7 out of 10 times) identify “something” leaking out through the client’s perimeter that they were unaware of. I thought I would compile a list of the top 5 so folks could sanity check their own security.

Firewall Threat #5: Commercial VPN services

It amazes me how often employees will do an end run around perimeter security and setup their own VPN solution. I even see this on networks that already have an existing VPN solution in place. Could be the end user thinks the corporate solution is too restrictive. Could be they simply want to be able to extract info without going through the corporate content checking solution. Unfortunately there are a number of third party companies that are more than happy to sell VPN services that will circumvent the average firewall policy.

How does it work?

There are a number of VPN service vendors, but the most popular is GoToMyPC (now owned by Citrix), so I will talk about that service specifically. Implementation for the end user is pretty straightforward:

• Create an account and pay the $20/month fee
• Load agent software on their system at work
• Go home and launch their Web browser

The user can then logon to their work system and control their desktop remotely. They can even transfer file information by dragging files in or out of the session window.

Why are third party VPN services a problem?

Third party VPN services allow your end users to create an encrypted tunnel between their work system and systems on the Internet. If you have implemented some form of Data Leak Prevention (DLP) or content checking on the perimeter to control the flow of company private information, they will be defeated by a third party VPN. This means your end users could be transferring anything out of your network. Not only do you have no control of what info passes through the VPN tunnel, these services try very hard to look like normal traffic so you may not even know this is happening on your network.

Under the hood

Figure #1 shows how a third party VPN solution can work. Note that session establishment originates from the work system, not from a system on the Internet. So as far as the firewall is concerned this is an outbound session. Further, many of these services use TCP/443 to communicate and SSL to secure the session. So from the firewall this looks like normal HTTPS traffic. Most admins do not log outbound HTTPS, so there may not be any log entries to review.

How to prevent it

There are a number of possible solutions to curb this activity:

1) Review outbound firewall logs

Specifically, you want to look at outbound traffic taking place after hours. Filter out patching sites (Microsoft, Adobe, etc.), A/V signature updates, or any other expected pattern. Everything else should be scrutinized. Pay close attention to repeated connection attempts to the same host or subnet at a predictable interval (say every 20-60 seconds).

2) Desktop enforcement

Leverage application control or a similar technology to control which applications your users can install on their desktop. While this option may seem to be the most cumbersome, its also the only one that will work most consistently.

3) Block the IP addresses of known VPN services

While this option will work, it requires research to find out which IPs you should ban. Also, the list is going to change as new vendors come on the scene or IPs get moved around. For these reasons its my least favorite option.

Firewall Threat #4: Secure Shell (SSH)

It pains me to put SSH on this list, as over the years I’ve found it to be one of my most useful tools. Unfortunately SSH can be your worst nightmare when its in the hands of your end users.

How does it work?

Most people know you can use SSH as a secure replacement for Telnet. What is not as well known is that SSH can be used to tunnel TCP traffic. This can be implemented as either a forward tunnel, or a reverse tunnel.

SSH forward tunnels

When a user creates an SSH forward tunnel, the SSH client will open up which ever TCP port is specified by the user. Further, the user can instruct the remote SSH server where to send any traffic sent to this local port.

An example is shown in Figure #2. Let’s say that all outbound Web traffic is sent through a content checking system. The firewall checks outbound HTTP to ensure that only appropriate sites are accessed and that certain keywords are not permitted through. Let’s further assume that we have an internal user who wishes to access inappropriate sites.

Here’s what the end user would do. On their home system they would run a proxy server as well as an SSH server. From their office system they would run an SSH client in order to connect to their SSH server and request a forward tunnel. They would tell the SSH client to open a local port (say 8080) and tell the SSH server to forward all of this traffic to the local proxy server. Now all they need to do is tell their Web browser to use the proxy server located at Localhost, port 8080.

Voila! When they launch their Web browser it will connect to TCP/8080 on the local system. This traffic is then sent down the encrypted tunnel to the SSH server, and then on to the proxy. The proxy server will then connect to what ever site on the Internet the user specified. The firewall can not content check the passing data or control which sites get accessed, because the datastream is encrypted. Our perimeter security has been circumvented.

By the way, the user does not have to limit access to only their personal system. Its possible to tell the SSH client to service connections for other hosts on the wire as well. I remember an on-site I once performed where my porn signatures were triggering on the internal network but not at the perimeter. Turned out one of the engineers had done the above and opened up access to their buddies. So when another engineer connected to his system, the porn content was visible. When it left his system and went to the Internet, it was now encrypted and protected from detection.

SSH reverse tunnels

SSH reverse tunnels are similar to SSH forward tunnels, but are designed to handle data requests headed in the other direction. This is shown in Figure #3. Let’s say we have an internal server which is only accessible to internal employees. In other words, the firewall does not permit inbound connection requests from the Internet to be forwarded to the server. In fact there may not even be any one-to-one NAT or port forwarding in place for folks on the Internet to even try and access the server. Let’s further assume that we have an end user who wishes to expose this server to the Internet.

This time the user only needs the SSH server on their home system. From their office system, they launch an SSH session out to their server and request a reverse tunnel. With a reverse tunnel the user tells the SSH server to listen on some local TCP port (say TCP/80) and send any inbound connection requests to the SSH client. The user then tells the SSH client to forward these connection attempts to a specific port at a specific IP address (say TCP/80 on an internal HR Web sever).

Once this connection is complete, anyone connecting to TCP/80 on the users home system will be forwarded to TCP/80 on the internal HR Web sever. The firewall can not control the session because it looks like outbound traffic and all the data is encrypted. In fact you can’t even detect this activity on the HR Web server as all connection requests will be logged as originating from the user’s internal desktop system (the one running the SSH client).

How to prevent it

Take control of all outbound SSH activity. Block TCP/22 outbound and only permit it through when its verified the traffic meets corporate policy. SSH can be configured to listen on any TCP port, so we have to be able to spot non-standard port use as well. Leverage a NIDS or NIPS signature that checks the first three incoming TCP reply packets for an SSH server banner. This should be done on all TCP ports except 22. The banner will contain the string “SSH-1” or “SSH-2”. While its possible to hack the client and server software to change the banners, most end users do not have necessary skills to modify the software and still have it remain functional.

That’s enough for this post. In the next I’ll include the last three threats.

C