Archive for August, 2009

« Older Entries

Spoofing Your IP Address During A Port Scan – Part 2

August 31st, 2009

In my last post I discussed an idle scan and how it can permit an attacker to mask their IP address during a port scan. In this installment we’ll look at some traces, as well as discuss how to identify when an idle scan has been used against your network.

Monitoring the IP ID increment

Let’s start by looking at the packets that were monitoring the IP ID field on the Windows system. Here is what our probe packet looked like:

07:22:15.367140 IP (tos 0×0, ttl 64, id 63897, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.1.2203 > 192.168.100.2.0: ., cksum 0xeca2 (correct), win 512

A few of these fields are kind of interesting. The TTL value is set to “64”, which suggests a Linux or UNIX system. Since we are raw writing the packet directly to the wire this value is actually controlled by hping. 64 just happen to be the program default, but we could change the value to anything we wish with the “-t” switch.

The target address is printed as “192.168.100.2.0”, which means the packet was sent to TCP port 0 at IP address 192.168.100.2. Windows does not offer services on TCP port 0 but that’s OK. We’re not actually trying to connect to the Windows system. We just need it to send us an IP packet so we can check the IP ID value. If we wanted to hit a different port, we could use hping’s “-p” switch.

The “.,” after the target specification means that no TCP flags were set within the packet. This is referred to as a null packet and would never occur during normal IP communications. For this reason, may NIDS, NIPS and firewalls will flag it. We created a null packet because we did not tell hping to sent any of the TCP flags. For example adding the “-S” switch would have turned on the SYN flag, “-A” would turn on the acknowledgment flag, and so on.

All of our probe packets to the Windows systems would look similar to this one. The only values that would change are the time stamp (obviously), the TCP source port and the CRC checksum value.

Here’s what the responses look like coming back from the Windows system:

07:22:15.367296 IP (tos 0×0, ttl 128, id 108, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.2.0 > 192.168.100.1.2203: R, cksum 0xc431 (correct), 0:0(0) ack 918250228 win 0

 

07:22:16.367453 IP (tos 0×0, ttl 128, id 109, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.2.0 > 192.168.100.1.2204: R, cksum 0xfa78 (correct), 0:0(0) ack 2127488152 win 0

 

07:22:17.367763 IP (tos 0×0, ttl 128, id 110, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.2.0 > 192.168.100.1.2205: R, cksum 0x2b9f (correct), 0:0(0) ack 1611256374 win 0

The important value here is the IP ID, identified as “id”. In the first packet it is set to 108, and then the value increments by +1 for each subsequent packet (109 then 110). As long as we continue to see an uninterrupted sequence in the IP ID, we know the Windows system is not transmitting any other packets except the ones that are part of this session.

Probing a closed port

Remember that as part of our scan, we will be spoofing the source IP address of the Windows system when target ports on a remote system. A change in the IP ID increment is what tells us we found an open port.

Let’s take a look at one of these can packets:

10:30:28.852602 IP (tos 0×0, ttl 64, id 41256, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.2.1266 > 192.168.100.3.79: S, cksum 0x97a6 (correct), 1704542340:1704542340(0) win 512

Note the source IP address is that of the Windows system (192.168.100.2). We know this could not have come from the Windows system however because the TTL is set to 64. So this is a packet we generated from 192.168.100.1 using a second instance of hping.

Also note we have targeted TCP port 79 and have turned on the SYN flag. Here is the response we received from the remote target:

10:30:28.852839 IP (tos 0×0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40) 192.168.100.3.79 > 192.168.100.2.1266: R, cksum 0xbb1a (correct), 0:0(0) ack 1704542341 win 0

Note the “R” value that tells us the reset flag was turned on in the TCP header. We also see “ack” which tells us the acknowledgement flag was turned on as well. This is an error packet informing the source that the TCP port is in a closed state. Also note the packet is being transmitted to the Windows system since that was the source IP address in the probe packet.

So how will the Windows system respond? You may remember from my last post that the RFCs state you should never respond to an error packet. Even though the Windows system has no idea what the target is talking about, it will quietly discard this error packet. In other words, the Windows system WILL NOT send a response. This means we should see no change in the IP ID increment.

Probing an open port

Now let’s take a look at what happens when we target a port that is actually in an open state. Here’s the probe packet. Note it is pretty similar to the last one, except now we are checking TCP port 80.

10:29:46.947964 IP (tos 0×0, ttl 64, id 15249, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.2.1984 > 192.168.100.3.80: S, cksum 0x476b (correct), 947341260:947341260(0) win 512

Here is the response from the target. Note that we see an “S” instead of an “R” which means they SYN flag is turned on rather than the reset flag. This is the targets way of informing the source that TCP port 80 is open and has an application sitting behind it servicing connections.

10:29:46.953333 IP (tos 0×0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 44) 192.168.100.3.80 > 192.168.100.2.1984: S, cksum 0x0de6 (correct), 262246932:262246932(0) ack 947341261 win 5840 <mss 1460>

So this time the Windows system receives a packet saying “Sure, you can connect to TCP port 80”. This is a problem for the Windows system because it has no connection entry saying that it actually tried to connect to port 80 on the target. Without the connection entry, Windows has no way of knowing which application requested the session. With this in mind, it does the only thing it can do; transmit a reset packet to the target killing the session. Here is what the packet looked like:

10:29:46.953439 IP (tos 0×0, ttl 128, id 176, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.2.1984 > 192.168.100.3.80: R, cksum 0x5df1 (correct), 947341261:947341261(0) win 0

Note that Windows stamped the next available IP ID in the IP header (176). So had we been monitoring the IP ID increment on the Windows system in a different session, we would see: 173, 174, 175, 177, 178, 179. In other words, we would have seen that 176 was missing (+2 from the previous packet). This would be our indication that the TCP/80 probe sent to the target hit an open port.

Identifying an idle scan

So an idle scan does an excellent job of masking the true source IP address of a port scan. This means that without access to log information on the Windows host (something that will not exist if the attack has done their homework), we will never be able to discover the true source IP address of the attack.

Is there a way to tell an idle scan was performed rather than a straight forward port scan so we know if it is worth investigating the source IP address? There are in fact a couple of clues we can look for.

Probe retries

If the targeted system is sitting behind a firewall, you will notice a difference in the number of probe retries vs. a normal port scanner. When someone runs a normal port scanner, the scanner will typically hit open ports only once but firewalled ports multiple times. This is because the firewalled ports return no response. The scanner will make multiple attempts to ensure the packets are not just getting lost. Since the open port will actually respond, only one probe packet is required.

When an idle scan is performed, the opposite is true. If a firewalled port is probed there will be no IP ID increment change on the Windows system. So only one check is required to confirm this is not an active listening port. When an open port is probed however, we’ll detect a change in the Windows system’s IP ID increment. While we think this is due to us finding an open port, it could just as likely be because the Windows system sent a broadcast. So we will want to perform multiple probes against the open port to ensure the Windows IP ID data remains constant.

So:

  • Probe firewalled ports more often than open ports = normal port scanner
  • Probe open ports more often that firewalled ports = idle scan

Timing

Normal port scans be extremely fast. It is not uncommon to see hundreds of probe packets per second. With an idle scan however, we have to take things a bit slower. This is because we must check the IP ID of the spoofed address, send the probe packet, permit enough time for the probe to elicit a response, give the spoofed system enough time to respond with a reset if required (thus using up an IP ID), and then check the IP ID again to see if the increment has changed.

Try to perform all of these steps too quickly, and you will pay the price in accuracy.  For example nmap supports idle scans with the “-sI” switch. The default extremely fast scanning speed works fine if all of the systems are on the same switched network. If the hosts are 15 hops away from each other on the Internet however, my experience has been the accuracy rating drops off considerably unless you slow down the scanning speed.

So:

  • Hundreds of packets per second = normal port scanner
  • A dozen or less packets per second = possible idle scan, could just be a slow scan

TTL field

Let’s look again at the spoofed probe packet as well as the legitimate reset response sent by the Windows system:

10:29:46.947964 IP (tos 0×0, ttl 64, id 15249, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.2.1984 > 192.168.100.3.80: S, cksum 0x476b (correct), 947341260:947341260(0) win 512

 

10:29:46.953439 IP (tos 0×0, ttl 128, id 176, offset 0, flags [none], proto TCP (6), length 40) 192.168.100.2.1984 > 192.168.100.3.80: R, cksum 0x5df1 (correct), 947341261:947341261(0) win 0

You may have noticed the TTLs are way off from each other. If both of these packets had in fact originated from 192.168.200.2 then they would both have the same starting TTL value. The fact that they are different tells us that one of them is spoofed.

If you are monitoring the perimeter of the target system and notice that the SYN and reset packets have different TTL values, that’s a clue that you may be experiencing an idle scan. Now, as mentioned earlier, it is entirely possible to set the TTL to what ever we wish in our spoofed packet. So a smart attacker is simply going to set their starting TTL to 128 to better mimic the Windows system. If they do this, your only hope is that the attacker and the Windows system are a different number of hops away. In other words, if you consistently see the SYN packets with a TTL of 115, but the resets consistently have a TTL of 113, you are most likely looking at an idle scan.

If the TTLs do match, we can’t entirely rule out an idle scan. The attacker may just be that good.

So:

  • TTLs of SYNs and RSTs are consistent but do not match = idle scan
  • TTLS of SYNs and RSTs match = normal port scanner or smart idle scanner

IP ID value

The best way to tag an idle scan is leverage the same thing the attacker uses, namely the IP ID value. Take another look at those last two decoded packets. Note that the IP ID in the first one is 15249 but in the second it is 176. If you are seeing an idle scan, you will notice the rest packets consistently have an IP ID of +2 (same thing the attacker sees), but the SYN packets will have some completely unrelated value. The IP ID increment in the SYN packets might be predictable or it might be random. Quite honestly it does not matter. What does matter is that you can spot some pattern in IP ID of the rest packets that does not jive with the IP IDs in the SYN packets.

So:

  • IP ID of SYN and RST packets appear related = normal port scanner
  • IP ID increment of RST does not match SYNs = idle scan

Exec Summary

Hopefully you now have a better understanding of what is happening on the wire when an attacker performs an idle scan. If we are targeted with an idle scan, we cannot determine the true source IP address of the attacker. The best we can hope to accomplish is to determine that the scan is an idle scan vs. a normal port scan. There are a number of tell tale clues in packets, the best of which is the IP ID value.

No comments »

Posted in 2-alert, Network Security, Packet Decoding

Tags:

Spoofing Your IP Address During A Port Scan – Part 1

August 28th, 2009

I love debunking myths, one of my favorites is “a port scanner must reveal his true source IP address”. In this series I’ll show you how to perform a port scan while hiding your source IP address from the host being scanned. I’ll also tell you how you can detect the technique when it is used against you.

Nmap’s decoy mode

An alternative to the technique I will describe is nmap’s decoy mode. With decoy mode you identify a number of bogus source IP addresses. From the target host, it looks like all of the bogus IP addresses, as well as the true source IP address, are all performing a port scan at the same time. The concept is the administrator under attack will have no way of knowing which IP address is in fact the true IP performing the scan.

This technique really does not mask the true source, as the source IP address is one of the IPs performing the scan. If you know what to look for, you can easily figure out which source IP is actually scanning you. So while this technique will work, it is not completely effective at hiding the source IP address.

What is an idle scan?

When we perform an idle scan, we do not actually directly detect open ports. Rather, we detect the effect an open port would have on a third party system. The technique is similar to how many viruses are detected in the human body. Rather than detecting the actual virus, we look for antibodies that get produced when the virus is present in the system. An idle scan detects open ports in much the same fashion.

Before we can dig too deeply into an idle scan, we need to look at some of the intricacies of IP.

Predictable header values

While the RFCs are designed to be specific enough that dissimilar operating systems will still be able to communicate via IP, they still leave quite a bit open to interpretation. For example, the RFCs specify that the maximum Time To Live (TTL) value that can be used is 255. They do not however specify what initial TTL value must be used; so different operating systems use different starting TTLs. The RFCs describe how Ping should work, but do not specify what should be in the payload of Echo-Request packets. Again, different vendors use different values. These nuances can permit you to identify the source operating system based variations in the packet contents. The technique is referred to as passive fingerprinting.

The IP identifier (IP ID) field in the IP header (bytes 4 and 5) is a similar situation.  RFC 791 specifies that the number must be unique on a per host, per session basis. For example let’s say I connect to a remote SSH server. Each IP ID in that session must be unique. If I close the session and then connect back later, it is RFC compliant if one or more IP ID values get used again. They don’t have to be, but if it does happen it is not a problem.

So the RFCs say the IP ID needs to be unique, but it does not really tie down how to go about generating the value. This has lead to different operating systems deploying different methodologies. For example Windows starts at an IP ID value of 1 and simply increments the value by +1 for every packet leaving the system. When the maximum value of 65,535 is reached, it starts back over at 1. BSD puts a random value into the IP ID field of each packet leaving the system. Linux is random for TCP packet (except initial responses which are always zero), +1 incremental for ICMP, and time based for UDP. Whew!

The one that is interesting for our purposes is Windows. The fact that each packet leaving the system gets a +1 IP ID makes the value extremely predictable. For example, consider the following output:

[root@fubar ~]# hping -r 192.168.100.2

HPING 192.168.100.2 (eth0 192.168.100.2): NO FLAGS are set, 40 headers + 0 data bytes

len=46 ip=192.168.100.2 ttl=128 id=108 sport=0 flags=RA seq=0 win=0 rtt=0.4 ms

len=46 ip=192.168.100.2 ttl=128 id=+1 sport=0 flags=RA seq=1 win=0 rtt=0.4 ms

len=46 ip=192.168.100.2 ttl=128 id=+1 sport=0 flags=RA seq=2 win=0 rtt=0.4 ms

len=46 ip=192.168.100.2 ttl=128 id=+2 sport=0 flags=RA seq=3 win=0 rtt=0.4 ms

len=46 ip=192.168.100.2 ttl=128 id=+1 sport=0 flags=RA seq=4 win=0 rtt=0.4 ms

len=46 ip=192.168.100.2 ttl=128 id=+1 sport=0 flags=RA seq=5 win=0 rtt=0.4 ms

hping is a packet crafting tool which allows you create your own IP packets. In the above output we are using the “-r” switch to have hping monitoring the IP ID increment of a remote system. We know it is a Windows system, because Windows always uses a starting TTL of 128. Now look at the “id=” values. In the first line of output hping always prints out the absolute IP ID value used by the system. In this case here the value is 108. Each subsequent line then prints out the delta change from the previous packet. So in the second line the actual IP ID was 109, which is “+1” from the previous value of 108. The next packet had an IP ID of 110, which is “+1” from the previous IP ID value of 109.

Look closely at the fourth line of output. Note the delta change was “+2”. Since Windows uses sequential IP IDs, this tells us a packet we didn’t get to see just left the Windows system. We don’t know where it was going, but that’s OK. What’s important is that we can identify when the Windows system transmits and how many packets it sends out. For example had that line read “+5”, we would know that the Windows system transmitted four other packets since responding to our last probe.

Detecting open ports

So how can we leverage the predictable IP ID value of Windows for evil? One possibility is to turn the Windows system into an open port sensor. Here’s how we do it:

  1. Monitor the current IP ID being used by a Windows system. We should check the value at regular intervals over a relatively short period of time. Say once second.
  2. Find a target system we wish to port scan.
  3. While spoofing the source IP address of the Windows system, send a SYN packet to the TCP port we wish to probe on the target.

The target system will send a response packet back to the Windows system.  This response will either be:

The RFCs state you should never respond to error packets, regardless of whether you consider them to be legitimate or not. So when the Windows box receives the TCP reset error packet from the target host, it quietly ignores and discards the packet.

Things get a bit more interesting when a SYN/ACK is received however. From the Windows system’s perspective, it is just hanging out minding it’s own business when some unknown system sends it a SYN/ACK packet (remember we spoofed the Windows system’s IP address in the probe packet). A SYN/ACK effectively means “Sure, you can connect to me on that TCP port, no problem”. Of course since the Windows system didn’t actually send the SYN packet, it has no idea what the remote target is talking about.

With this in mind the Windows system sends a TCP reset error packet back to the target host. When the reset packet is transmitted, the next available IP ID is used within the IP header. This missing IP ID would be detected if we are still monitoring the IP ID increment once per second. So to review:

  • Closed port on target = No packets leaving Windows system
  • Open port on target = Windows sends a TCP reset using up an IP ID

So by monitoring the IP ID increment, we can identify when an open port is discovered as only probes to open ports will cause the IP ID increment to change.

Caveats

You can’t use just any Windows system for this attack. The box must meet certain criteria:

  • Relatively quite system generating little traffic (like a home system)
  • No stateful filtering of TCP traffic

Of course go to any cable or DSL network at 2:00 AM local time and you can find hundreds of thousands of systems that meet these criteria. Remember that Windows systems love to arbitrarily broadcast, so you may wish to perform multiple check of each open port just to ensure the IP ID increment change was in fact due to an open port being probed.

Exec Summary

An idle scan lets you probe open ports on a remote target, while fooling the target into believing that some third party system is performing the scan. Open ports are detected by monitoring for irregularities in the IP ID increment of the Windows box.

In the next installment we’ll actually see what these packets look like on the wire as well as discuss how to detect an idle attack when it is used against you.

No comments »

Posted in 3-err, Network Security, Windows Security

Tags:

Network Mapping Through A Firewall – Part 3

August 26th, 2009

In my last two posts I talked about two different methods that can be used to map a network through a firewall. The first leveraged ICMP time exceeded in transit errors, while the second used the IP record route option. In both posts I also gave possible solutions for preventing an attacker from using these techniques against your network.

In both cases however, supported features available in commercial grade firewalls limited our security options. In this third and final part of the series, I will cover how to properly prevent these attacks if you are using an open source firewall. I will specifically be using Netfilter, but many of the techniques are applicable to pf as well.

What is Netfilter?

Netfilter is the stateful inspection firewall that is included in every modern distribution of Linux. If you have a copy of Linux, you also have a copy of Netfilter. Netfilter is sometimes referred to as iptables, but this is because iptables is the name of the binary you use to manipulate the Netfilter rulebase. Netfilter is an extremely capable firewall with too many features to cover in this post. I highly recommend you check out some of the FAQs and tutorials as they do an excellent job of describing many of the features.

Controlling tcptraceroute

In the first post I described how tools like tcptraceroute could punch through an open firewall rule to map the network sitting behind it. With commercial firewalls, we were limited to controlling the flow of outbound ICMP time exceeded in transit errors.

With Netfilter, we have the ability to control traffic based on the TTL value. We can look for a specific value, or a value above or below a certain threshold. The supported switches are:

  • -m ttl –ttl-eq = Match packets with a TTL of a specified value
  • -m ttl –ttl-gt = Math packets with a TTL higher than a specified value
  • -m ttl –ttl-lt = Match packets with a TTL below a specified value

Here is a possible Netfilter rule we can use:

iptables -A FORWARD -m ttl –ttl-lt 5 -j DROP

This rule would be processed prior to any permit rules in the rulebase. The rule simply checks the TTL value to see if it is less than 5. If so, the packet is dropped. Since the lowest TTL used by a modern OS is 64, and most systems are about 15 hops away from each other on the Internet, we should never inadvertently filter out legitimate traffic.

Here’s tcptraceroute running though a regular firewall:

[root@fubar ~]# tcptraceroute -n -f 1 -m 5 -q 1 -S 10.1.4.10 80
Selected device eth0, address 10.1.1.10, port 39142 for outgoing packets
Tracing the path to 10.1.4.10 on TCP port 80 (http), 5 hops max
1 10.1.2.2 0.353 ms
2 10.1.4.1 0.450 ms
3 10.1.4.10 [open] 0.586 ms

And here is what tcptraceroute sees once we implement the above Netfilter rule:

[root@fubar ~]# tcptraceroute -n -f 1 -q 1 -S 10.1.4.10 80
Selected device eth0, address 10.1.1.10, port 54531 for outgoing packets
Tracing the path to 10.1.4.10 on TCP port 80 (http), 30 hops max
1 10.1.2.2 10.175 ms
2 10.1.4.1 0.464 ms
3 *
4 *
5 *
6 *
7 10.1.4.10 [open] 1.007 ms

Note that once we start filtering on TTL value, the appearance of our perimeter changes. Without the rule an attacker could enumerate or IP addressing scheme. Even if we filtered outbound TimeX packets, they would still know the proper hop count. The Netfilter rule makes it much more difficult to accurately identify our network layout.

Adding in some deception

One of the more powerful features of Netfilter is the ability to customize reject messages. While most firewalls reject packets by returning an administratively prohibited error message, Netfilter lets you choose from a number of different unreachable error codes. This makes for some interesting possibilities. For example, consider the following rule:

iptables -A FORWARD -m ttl –ttl-lt 5 -j REJECT –reject-with icmp-host-unreachable

This rule tells Netfilter that whenever it sees a packet with a TTL less than 5, it should return an ICMP destination host unreachable packet. In other words, Netfilter will impersonate a router and tell the transmitting system that the target host is off-line. Here’s an example of tcptraceroute output once this rule has been implemented:

[root@fubar ~]# tcptraceroute -n -f 1 -q 1 -S 10.1.4.10 80
Selected device eth0, address 10.1.1.10, port 47555 for outgoing packets
Tracing the path to 10.1.4.10 on TCP port 80 (http), 30 hops max
1 10.1.2.2 0.299 ms
2 10.1.4.1 0.450 ms
3 10.1.4.1 0.403 ms !H

Compare this output to the first tcptraceroute output shown above. Note that line 3 is now different. With a regular firewall, hop three was a response from the target host. In this output however, it appears the upstream router is returning an ICMP host unreachable (designated as “!H”) signifying the host is off-line. Since tcptraceroute thinks the host is off-line, it gives up trying and never actually reaches the target host.

So while this technique is a bit of security through obscurity, it is effective at disabling a tool that would normally punch right through a firewall. Since regular traffic would not have an abnormally low TTL value, it does not match this rule and is unaffected.

Controlling record route

In my second post in this series I talked about record route and how it can be leveraged to map through a firewall. I discussed that the range of the tool is limited (max 8 hops, 3 if you want hop info in both directions), but that there are ways for an attacker to get around this restriction. I also mentioned that commercial firewalls typically do not give you the ability to control record route traffic.

With Netfilter, there is support for controlling IP options via the ipv4options module (http://netfilter.org/documentation/HOWTO/netfilter-extensions-HOWTO-3.html). The supported switches are:

  • -m ipv4options –ssrr = Match packets with strict source routing set
  • -m ipv4options –lsrr = Match packets with loose source routing set
  • -m ipv4options –rr = Match packets with record route set
  • -m ipv4options –ts = Match packets with timestamp set
  • -m ipv4options –ra = Match packets with router-alert set
  • -m ipv4options –any-opt = Match packets with at least one IP option set

Here’s an example of a rule that would block packets with the source route option set:

iptables -A FORWARD -m ipv4options –rr -j REJECT –reject-with icmp-host-unreachable

Note we are sending back an ICMP host unreachable in response. This is in order to shutdown the tool mapping our network.

Exec Summary

While commercial firewall excel at centralized management and selecting pleasing colors for their graphical interface, they usually pale in comparison to open source firewalls with regards to controlling traffic on the wire. In order to protect their networks, firewall administrators need greater control of the IP header than simply scrutinizing the source and destination IP address.

No comments »

Posted in 2-alert, Network Security

Tags:

Network Mapping Through A Firewall – Part 2

August 25th, 2009

In my last post I discussed how to use ICMP time exceeded in transit errors to map a network perimeter. I also discussed how to prevent attackers from using this technique against your network. In this post I’ll discuss another network mapping technique using the record route IP header options.

Ipv4 header options

The IP header is normally 20 bytes in size but can grow larger if one or more options are enabled. IP options get added to the end of the IP header, as shown in Figure #1. There are a number of registered IP options. The ones most frequently implemented however are the ones defined in RFC 791. Most operating systems and hardware devices have implemented the IP option record route (option 7), which is a part of the RFC 791 specification.

IP-Header-options

Record Route

The record route option can produce similar data to traceroute, but has a completely different methodology for identifying intermediary hops. As I discussed in my last post, traceroute uses the receipt of ICMP time exceeded in transit errors to map all of the network hops between two points. This requires multiple packets to be transmitted, as the tool needs to increment the TTL value.

Record route does not vary the TTL, and only requires a single packet to record hops along a link. Since the option exists within the IP header, it can be leverage with any IP transport or application.

Here is example output of a record route session using Ping under Linux:

[root@fubar ~]# ping -c 1 -R 192.168.204.10

PING 192.168.204.10 (192.168.204.10) 56(124) bytes of data.

From 192.168.201.1: icmp_seq=1 Redirect Host(New nexthop: 192.168.202.2)

64 bytes from 192.168.204.10: icmp_seq=1 ttl=125 time=6.56 ms

NOP

RR:       192.168.201.10

192.168.201.1

192.168.202.1

192.168.203.1

192.168.204.10

192.168.204.1

192.168.203.2

192.168.202.2

192.168.201.10

— 192.168.204.10 ping statistics —

1 packets transmitted, 1 received, 0% packet loss, time 6ms

rtt min/avg/max/mdev = 6.564/6.564/6.564/0.000 ms

Note that by setting the record route option in Ping (the “-R” switch) we’ve recorded all the router hops out to the target system at 192.168.204.10, and back again. So we’ve effectively generated a map of the network between the two points.

Record route decode

Here is an example decode of a record route packet:

07:04:32.934999 IP (tos 0×0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 124, options (NOP,RR 192.168.201.10, 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0)) 192.168.201.10 > 192.168.204.10: ICMP echo request, id 43604, seq 1, length 64

0×0000:  4f00 007c 0000 4000 4001 6858 c0a8 c90a  O..|..@.@.hX….

0×0010:  c0a8 cc0a 0107 2708 c0a8 c90a 0000 0000  ……’………

0×0020:  0000 0000 0000 0000 0000 0000 0000 0000  …………….

0×0030:  0000 0000 0000 0000 0000 0000 0800 5df6  …………..].

0×0040:  aa54 0001 4022 914a 2544 0e00 0809 0a0b  .T..@”.J%D……

0×0050:  0c0d 0e0f 1011 1213 1415 1617 1819 1a1b  …………….

0×0060:  1c1d 1e1f 2021 2223 2425 2627 2829 2a2b  …..!”#$%&’()*+

0×0070:  2c2d 2e2f 3031 3233 3435 3637               ,-./01234567

A couple of points in the above decode are worth noting. Normally the beginning of the IP header starts with a Hex value of 4500. This means:

  • 4 = IP version
  • 5 = 5 32-bit words, or (32/8) x 5 = 20 bytes, the size of the IP header
  • 00 = Type Of Service (TOS) field, no values set

The decode above starts with the Hex value “4f00”, which means the IP header is larger than a regular IP header. This is our first clue that at least one IP option is set. How big is the IP header? If we convert “f” in Hex to decimal we get 15. 15 32-bit words converts to 60 bytes, which is the largest possible size for an IP header.

Also, note the series of zeros at the end of the header. When a record route packet is transmitted, the sending system needs to reserve space for all of the IP addresses that must be included. Windows will ask you to identify this value up front. Linux and UNIX simply go for the maximum. It does not cause a problem if reserved space goes unused. The rest of the packet carries a normal Echo-Request payload.

Record route limitations

You may have noticed that the above decode only reserved space for 8 IP addresses. Since most systems on the Internet are about 15 hops away from each other, what happens when 8 is not enough? Remember we said 60 bytes is the maximum size for an IP header. If we remove the rest of the IP header fields, that leaves us enough room to store 9 IP addresses. The transmitting system always stores it’s IP address in the option field, since technically it is the first IP address to forward the packet. This leaves us enough room for 8 more IP addresses maximum. If the packet travels over more than 8 hops, the remaining routers will simply ignore the record route option.

Here’s an example of what I mean. This output was generated with the Ping utility under Windows. The “-r” switch identifies that the record route option should be set. The numeric value identifies how many hops to record.

C:\test>ping -r 8 -n 1 www.wikipedia.org

Pinging rr.pmtpa.wikimedia.org [208.80.152.2] with 32 bytes of data:

Reply from 208.80.152.2: bytes=32 time=702ms TTL=50

Route: 98.232.117.112 ->

68.88.131.63 ->

68.87.145.246 ->

68.87.145.245 ->

68.85.162.70 ->

68.86.90.65 ->

4.68.185.30 ->

4.69.132.90

Ping statistics for 208.80.152.2:

Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 702ms, Maximum = 702ms, Average = 702ms

The Wikipedia Web site is actually 19 hops away from my current location. Record route is only capable of recording the first 8 hops along the way.

Do I need to be concerned with record route?

Since record route is only capable of recording 8 hops, and most of us are 15 hops away from each other, is it truly a valid security concern? The 15 hop rule is only true a majority of the time. If I attempt to record route to a network that uses the same ISP that I do, I’ll probably generate a full network map. Further, if an internal system becomes compromised, record route can easily be leveraged to map the network from the compromised host’s location.

So record route is not a common attack vector, but its certainly going to be one of the tools a smart attacker will leverage when possible.

Protecting against record route

Record route is one of those communication parameters that gets ignored by most commercial firewall vendors. By that I mean they include support for record route in their RFC compliant IP stack, but give you little ability to control it via policy enforcement. Open source firewalls tend to do a better job controlling record route, but I’ll get into that in part 3 of this series.

If your firewall, HIPS or HIDS gives you access to the signature language, you can usually write a signature to flag all packets with an IP header size larger than 20 bytes. This does not guarantee the packet is using record route, as it could also mean that some other IP option is being used. To be frank however, all of the IP options can be leveraged for evil. Every one of them should be blocked, or at the very least detected, at the perimeter. I’ll cover more about IP options in a later post.

Exec Summary

Record route can produce a network map similar to the traceroute tool, but is limited to only recording 8 hops. While this limits its usefulness to an attacker, its entirely possible to run a record route session close enough to the target network to enumerate valuable data. Most firewalls do not give you the ability to control record route traffic, but you may be able to control/detect it with a signature based device.

No comments »

Posted in 3-err, Network Security

Tags:

Network Mapping Through A Firewall – Part 1

August 24th, 2009

When we create a set of firewall rules, one of our objectives is usually to stop attackers on the Internet from being able to map the internal network sitting behind the firewall. In this write up I’ll discuss two different techniques which will let an attacker punch right though most firewall setups, and what additional steps must be taken to prevent them.

The two techniques we will cover are:

  • Eliciting time exceeded in transit errors
  • IP header record route options

Understanding Time exceeded in transit errors

When a router receives a packet traveling from one network to another, it is required to decrement the TTL value by one. So if the packet currently has a TTL of 120, the router would change the value to 119 as it passes the packet along the network. The TTL field is byte 8 within the IP header and is shown in Figure #1.

IP-Header

If a router receives a packet with a TTL value of 1, it is not allowed to decrement the value to 0. Rather, the router generates an ICMP type 11, code 0 packet; referred to as an ICMP time exceeded in transit (TimeX) error. The TimeX error is then sent to the source IP address listed in the packet that had a TTL value of 1. Here’s an example TimeX packet. Note that 28 bytes of the original packet that caused the TimeX to be generated is embedded in the payload. The TTL value of this embedded header is 1.

10:14:19.947925 IP (tos 0xc0, ttl 63, id 26344, offset 0, flags [none], proto ICMP (1), length 88) 192.168.202.2 > 192.168.201.10: ICMP time exceeded in-transit, length 68
IP (tos 0×0, ttl 1, id 34730, offset 0, flags [none], proto ICMP (1), length 60) 192.168.201.10 > 192.168.204.10: ICMP echo request, id 18212, seq 1, length 40

One interesting point here is that RFC 792 defines that packets should be dropped when the TTL reaches 0, not 1. I’m unaware of any router or system that actually follows the RFCs. Every device I’ve seen drops the packet when the TTL is 1. You will however find many incorrect documents that describe this process quoting the RFCs rather than reality.

Network mapping with TimeX

Most network administrators are familiar with the traceroute and LFT tools under Linux and UNIX, and tracert and pathping under Windows. Each tool will identify all of the router hops from a source system to a specified target. This is accomplished by transmitting multiple packets and incrementing the TTL value.

Each of the above mentioned tools use TimeX errors to map all of the routers between two hosts. An example is shown in Figure #2. The tool would start by transmitted packets with an initial TTL value of 1. This causes the first router to return a TimeX error. The tool then looks at the source IP address of the TimeX error, and records this as the first hop along the link.

tracing

Packets with a TTL of 2 are then transmitted. When they pass through the first router, the TTL is decremented to 1. This causes the second router to generate a TimeX error. Again, we simply record the source IP address of the TimeX error as the second hop along the link. When an initial TTL value of 3 is transmitted, the third router generates the TimeX error. This continues until we eventually reach the target system. We’ve now efficiently mapped the IP addresses of all of the routers between the source and target system.

Here’s an example of what the output might look like:

[root@fubar ~]# traceroute -I -q 1 -N 1 10.1.4.10
traceroute to 10.1.4.10 (10.1.4.10), 30 hops max, 60 byte packets
1 10.1.1.1 (10.1.1.1) 0.270 ms
2 10.1.2.1 (10.1.2.1) 0.395 ms
3 10.1.3.1 (10.1.3.1) 0.589 ms
4 10.1.4.10 (10.1.4.10) 0.707 ms

Mapping through a firewall with time exceeded packets

The tools tracert and traceroute are easily defeated by a firewall. This is because tracert transmits Echo-Request packets which most environments block at the border. traceroute will also transmit Echo-Requests if the “-I” switch is used, but by default it targets UDP ports above 33,000. Again, most firewalls block this by default so the tool is easily defeated.

But what if an attacker targets an open port on the firewall? In other words, what if they transmit TCP/80 packets to your Web server, but vary the TTL values in a similar fashion to traceroute? This is exactly how the tool tcptraceroute operates. There is even a version available for Windows. Usually, tools like this can map right though a firewall.

For example, we have a Web server at 192.168.204.10 with a firewall sitting in front of it. The firewall has the standard “only let in TCP/80 to the Web server” policy set. Here is what traceroute reports:

[root@fubar ~]# traceroute -q 1 -N 1 -m 5 10.1.4.10
traceroute to 10.1.4.10 (10.1.4.10), 5 hops max, 60 byte packets
1 10.1.1.1 (10.1.2.2) 0.279 ms
2 10.1.2.1 (10.1.4.1) 0.521 ms
3 *
4 *
5 *

And here is the same networks mapped with tcptraceroute:

[root@fubar ~]# tcptraceroute -n -f 1 -m 5 -q 1 -S 10.1.4.10 80
Selected device eth0, address 10.1.1.10, port 39142 for outgoing packets
Tracing the path to 10.1.4.10 on TCP port 80 (http), 5 hops max
1 10.1.1.1 0.353 ms
2 10.1.2.1 0.450 ms
3 10.1.3.1 0.586 ms
4 10.1.4.10 [open] 0.701 ms

Because traceroute is sending UDP packets, our firewall policy drops them at the border. tcptraceroute however is sending TCP/80 packets to the Web server’s IP address. Since this is permitted by the policy, the packets make it through. We now know 10.1.3.1 is acting as a firewall. We also know that it is sitting directly in front of the Web server.

Here’s a copy of one of the packets generated by tcptraceroute. To the untrained eye, it looks like a perfectly normal TCP/80 SYN packet, except the TTL value is very low (there are other clues that this packet is not normal, but I’ll save that for another post):

18:33:21.531117 IP (tos 0×0, ttl 3, id 41587, offset 0, flags [none], proto TCP (6), length 40) 10.1.1.10.37496 > 10.1.4.10.80: S, cksum 0x7eaa (correct), 1793661553:1793661553(0) win 0

Protection against TimeX mapping

Most stateful inspection based firewalls are horrible at stopping TimeX mapping. In part 3 of this post, I’ll get into the proper way to control TimeX if you are running an open source firewall. For now however, I want to limit the advice I give to solutions that will work for every product.

There are two parts to every conversation, the stimulus and the response. When it comes to network mapping we can effectively nullify a scan if we can control either portion of the conversation. In this case here we have:

  • Stimulus = IP packet with an abnormally low TTL value
  • Response = TimeX from routers, port response from target

Since most commercial firewalls do not permit you to filter traffic based on TTL, we can’t control the stimulus in this situation. Nor can we control the port response, because it will be identical to a normal conversation. This leaves us with the outbound TimeX packets.

As close as possible to the edge of your perimeter, install a filter preventing ICMP type 11, code 0 (Time Exceeded in transit) packets from being sent to the Internet. For example if you have a border router outside of your firewall, install the filter on the router. Note that if you are running Cisco IOS, the router will partially ignore the filter and still transmit TimeX packets generated by it’s own interface. Running the “no ip unreachables” command can prevent this, but this command disables all ICMP error reporting and can cause communication problems. Make sure you understand the full impact of this command before using it.

By filtering outbound TimeX packets, we will prevent the attacker from seeing the IP address of all routers and firewalls between the filter installation point and the target host. The attacker will still be able to enumerate how many hops are on the link; they just will not be able to determine the IP address of each.

Exec Summary

Tools that perform traceroute type activity through open ports on a firewall are effective at mapping the links along a target network. Further, these tools are usually effective as enumeration of network address translation (NAT) settings. Since most firewalls cannot filter traffic based on TTL, we are usually left with trying to control the transmission of TimeX packets headed out towards the Internet.

No comments »

Posted in 3-err, Network Security

Tags:

Setting Up A Security Information Management System-Part 6

August 20th, 2009

So far in this series we have covered:

  • Defining a scope and focus for your SIM
  • Importance of building instead of buying your first system
  • Architecture and capacity planning
  • Recommended phases of deployment
  • Selecting a centralized logging server platform
  • How to accept remote log entries
  • Facility, severity and priority
  • How to sort log messages
  • Configuring appliances and operating systems to submit log entries

Cool. So we have log entries for a number of systems being collected on a centralized server. Now comes the most important task, leveraging that information. Log entries will be grouped into two categories; critical messages we want to know about right away, and log entries that will get caught as part of a regular review process.

Blacklisting Vs. Whitelisting

When reviewing log messages, we have two possible postures we can use. The first is referred to as blacklisting. With the blacklisting method we define what makes an event interesting enough to warrant reporting. This is similar to how anti-virus software detects Malware or the process we use to filter out spam.

Like most things in life, blacklisting has some good and bad aspects. On the plus side, it is usually pretty easy to write a signature if we know what we want to look for. Signatures can be tightly defined to help minimize the number of false positives we encounter. The problem with blacklisting is that we have to know what we are looking for. If a new attack generates a unique signature we have never encountered in the past, a blacklisting system will probably miss the event because no signature has been defined.

With whitelisting we define the events we understand, and then focus our attention on the new and unique log messages that are encountered. On the plus side we are far more likely to catch cutting edge attacks. Whitelisting tends to be relatively noisy however since we are bound to encounter unique log messages that are not indicative of a security event.

So which should we use? Good defense in-depth practices tell us to use both. ;)

Real time alerting

We can leverage blacklisting to perform real time alerting of event we want to be made aware of as soon as they occur. Blacklisting should only be used for low noise types of events. In other words, we want to stick with writing signatures for events that have a high probability of being a true security issue. Good examples are:

  • Different logon name failures all from the same IP address in a short amount of time
  • Multiple HTTP 403 errors being generated by a single IP in a short amount of time
  • Internal systems receiving many ICMP errors or TCP resets in a short amount of time

In order to perform real time alerting, we need software that will monitor the logs in real time. The log entries should be checked against defined signatures, which also indicate what to do when the event occurs.

Swatch

One of the easiest tools you can use for monitoring log entries is Swatch. Swatch is based on Perl. This means that while it is designed for UNIX and Linux systems, you can get it running on Windows if you have Perl installed. Simplicity is both Swatch’s biggest strength and weakness. While Swatch is relatively easy to deploy, it is also somewhat limited in its functionality. Still, if you are new to logging, Swatch makes an excellent first tool for real time alerting.

To deploy Swatch, you will need to create a unique configuration file for every log file you wish to monitor. In the configuration file we will tell Swatch what to look for in that particular log file, and what to do when the event is detected.

For example, let’s say we are going to have Swatch monitor the Web server’s error log. We may wish to create an entry similar to the following in Swatch’s configuration file for the error log:

# Look out for buffer overflows

watchfor  /client denied by server configuration|File name too long/

mail=noc@fubar.org:webmaster@fubar.org,subject=Web server overflow attempt

The line beginning with a “#” is simply commentary on the signature. The watchfor line identifies which character string(s) we want to define as being interesting. In this particular rule we have defined two different strings, “client denied by server configuration” and “File name too long” as interesting. The pipe character between the strings acts as a logical “or”. If either string is encountered, the mail parameter defines two different e-mail addresses we should contact. The subject line of the e-mail will be “Web server overflow attempt”, while the body of the e-mail will be the actual log entry.

If there are other patterns we wish to detect, we could add additional watchfor and mail statements. If we want to do more than send an e-mail, the exec parameter can be used to execute any application located on the local system. The threshold parameter can also be used to rate limit the reporting of events.

Simple Event Coordinator (SEC)

SEC is an amazing alerting tool you can download from the main Web site. It supports BSD and Linux, and ships with a number of popular Linux flavors. SEC fully supports regular expressions and allows you to create extremely granular signatures.

The rule format is as followed:

type= Method of detection

ptype= Pattern type (regular expression, string match)

pattern= What to search for

desc= Description (can be a variable)

action= What to do when detected

There is an excellent archive of pre-written rules you can use that is well worth looking at. You can match on multiple patterns, define multiple thresholds, all while processing hundreds of log messages per second. About the only drawback of SEC is that you need a good understanding of regular expressions to use the tool effectively. Still, the tool can be far more powerful and flexible than Swatch.

Where can I get more alerting ideas?

I was involved with the creation of the original SANS Top 5 Log Reports. For the April, 2009 Log Summit I updated my presentation to break up report examples into low noise and high noise categories. Anything on the low noise list would make a good candidate for alerting. Anything in the high noise section is better monitored through daily reports.

Daily Reports

So we leveraged blacklisting to generate our real time alerts. We’ll now leverage whitelisting to help highlight unknown but interesting traffic patterns within our daily reports.

When it comes to daily reports, we tend to gravitate towards the big numbers. What are the top 5 IPs transferring data? Which e-mail address sent the most messages? While the big numbers are certainly important, it has been my experience that the security events you need to worry about the most generate the fewest log entries. The smart attackers try very hard to remain hidden within the noise. So the only way to find them is to lower the signal to noise ratio.

I author the Perimeter Security track for SANS. One of the labs I have my students perform is to parse a 200,000 line log file. The goal is to spot the interesting patterns as well as formulate the review into an automated process. Most folks find the port scanner as it is pretty noisy. Some even spot the IP address performing application layer attacks against the Web server. What most people miss however, are the six lines that are a pretty clear indication that an internal system is already compromised and calling home for marching orders. How do you find those 6 lines? By whitelisting everything you understand and focusing on what ever is left.

So it is OK for our daily reports to give us pretty charts with big numbers. One of the reports however has to be able to move all the crud to the side so we can better spot the interesting patters.

Logwatch

One of the best tools for doing a daily log review is Logwatch. Logwatch will summarize all of the log patterns it understands, while highlighting anything without a predefined signature. Best way to understand this feature is to look at an example.

SSHD Killed: 2 Time(s)

SSHD Started: 1 Time(s)

Connections:

Failed logins from these:

msmith/password from 1.3.247.11: 6 time(s)

jsmith/password from 1.3.247.11: 5 time(s)

psmith/password from 1.3.247.11: 4 time(s)

Users logging in through sshd:

jjones logged in from sundown (1.3.247.9) using publickey: 146 Times(s)

jsmith logged in from dialup5533.wnskvtao.sover.net (216.114.181.200) using password: 1 Times(s)

jsmith logged in from dialup984.wnskvtao.sover.net (216.114.163.223) using password: 1 Times(s)

bjones logged in from charlie (1.3.247.11) using publickey: 444 Times(s)

jsmith logged in from 192.168.1.173 using password: 2 Times(s)

djones logged in from charlie (1.3.247.11) using password: 47 Times(s)

**Unmatched Entries**

Received disconnect from 148.64.147.168: 3: Key exchange failed.

Received disconnect from 216.114.160.132: 11: All open channels closed

scanned from 146.87.114.150 with SSH-1.0-SSH_Version_Mapper.  Don’t panic.

scanned from 211.184.226.99 with SSH-1.0-SSH_Version_Mapper.  Don’t panic.

In the above example Logwatch is being used to summarize SSH activity. It understands the service being stopped and started, failed logon attempts as well as successful logons. All this information is displayed in summary format so it is easier to digest. For example we do not know exactly when msmith incorrectly entered their password, but we see it happened six times, all from IP address 1.3.247.11. So instead of having six lines to digest, we only need to look at one. If we want to see each specific log entry, we can always refer back to the original logs.

Now look at the “Unmatched Entries” section. Each of these is an event that Logwatch does not have a signature for. Rather than ignore them, which would happen with a blacklist based system, they are summarized here for us to review. We then have the option to generate a signature for a specific entry so it will get categorized in a similar fashion to the process and logon sections.

Clearly this gives us the best of both worlds. The above report represents a bit over 650 lines worth of log entries, summarized down into an easy to read report. Most importantly, none of the log entries had to be ignored in order to produce this summary report.

Beyond daily reporting

You may also find it useful to perform long-term trend analysis and data mining on your log data. This may help to reveal patterns that normally go undetected when logs for a small snapshot in time (like 24 hours) is reviewed. Arguably one of the best tools available for dealing with lots of data is Splunk.

Splunk

Splunk is available as a free version that is limited to processing 500 MB per day, or you can invest in the commercial version that supports unlimited data processing. Splunk is extremely flexible at accepting data. It can act as a centralized logging server, or you can transfer files via a number of methods including FTP and HTTP. Once the data is received, Splunk indexes every field in each log file. This gives you unparalleled sorting and searching capability.

The full features are Splunk are too numerous to get into in this post. Check their site for a full list of supported features. What Splunk is extremely good at is manipulating and reporting on a huge number of log entries. It can index, search and report on billions of log entries per second. This makes it extremely useful for generating long-term trend reports or running saved searches for data mining purposes.

Exec Summary

We’ll we’ve reached the end of the trail. Hopefully you feel like you have a better handle on how to deploy a centralized logging solution, as well as how to leverage it to better secure your environment. If you have any questions, please feel free to drop a comment. :)

No comments »

Posted in 3-err, Logging

Tags:

Why anti-virus is dead – live

August 18th, 2009

The keynote I did at the Proactive Cyber Defense Seminar seems to have been well received. I’m going to be doing an abridged version of the talk as a Webinar on August 27th. If you feel like you are fighting a losing battle with Malware, you may find the talk insightful. Sign up is free and more info can be found on the Live page.

No comments »

Posted in 5-info, Malware

Breaking Out Of A sudo Shell

August 18th, 2009

I’ve been meaning to write this up for a while, but had a post go by on one of the SecurityFocus mailing lists that motivated me to get it done. The post asked, “How do you restrict sudo so a user can not delete the whole hard drive?”. The short answer is, it is extremely difficult.

Sudo is a security tool used within the UNIX and Linux world. It permits the Administrator to restrict access to high-level commands. It differs from the su command in that su immediately gives the user root level access. Sudo permits the Administrator to fine-tune which root commands can actually be accessed. In addition, sudo creates an audit trail of all executed commands.

The problem with sudo is that it has no visibility when a tool permits a command shell to be spawned. This means a user can misbehave on the system while generating an audit trail, which makes it appear they have done nothing wrong.

Try it yourself

I’m going to walk you through an example so you can experience exactly what I mean. You will need a Linux system for this. If you don’t have one handy, Backtrack should work just fine. The test system I’m using is based on Fedora but they are pretty similar.

Step #1: Add yourself to the users group

Use your favorite text editor to edit the /etc/group file. Look for the “users” group and add your account name. In this example I’ve added the account name “cbrenton”.

users:x:100:cbrenton

Step #2: Create a sudousers entry

Next we need to edit the /etc/sudoers file to define what the users group is permitted to do. We need to use visudo for this, which is a text editor based on the vi editor. If you are comfortable with vi, type in the command “visudo” and add this line to the end of the file:

%users ALL=/usr/bin/less

If you do not know vi, I’ll give you the exact commands. Remember everything is case sensitive:

visudo

G

o        <– lower case letter O

%users ALL=/usr/bin/less

<Esc><Esc>

:wq

Step 3: Monitor sudo activity

Open a second terminal window and use “su -” to assume root privileges. Type in the command:

tail -f /var/log/secure

This will permit us to monitor all sudo activity.

Step 4: Run a sudo command

Back in the first terminal window type in the command:

sudo less /etc/hosts

You will be prompted for your password (your password, not root’s password). Once you type in your password you should see the contents of the /etc/hosts file. In the second terminal window you should now have a log entry similar to the following:

Aug 18 16:07:58 fubar sudo: cbrenton : TTY=pts/4 ; PWD=/home/cbrenton ;USER=root ; COMMAND=/usr/bin/less /etc/hosts

So we can see date/time, who executed the command, which terminal they were on, what directory they were in, what permissions level they were using, and what command they executed including any command line switches (in this case a file specification). So far, so good.

Step 5: Bust a shell

In the first terminal window (running the less command) type in:

! <Enter>

You should now be looking at a root command prompt. To verify you are in fact root, type in the command:

whoami

Note that typing in the command “whoami” did not generate a security log entry. In other words, once we spawned a command shell with the less command, we can do anything we want and sudo will not log it.

Step 6: Additional verification

In the first terminal window where “less” is in the background, type in the command:

top

This will print out statistics about the running system.

In the second terminal window note the terminal in use. In the above example log entry the terminal is “TTY=pts/4″ but yours might be different. Make a note of the terminal. Now, in this second window press:

<CTRL>-C

to kill the tail command. Now run the command:

ps ax | grep ‘pts/4′

Replacing “pts/4″ with whatever terminal name you noted. You should receive output similar to the following:

17330 pts/4    Ss     0:00 bash

18392 pts/4    S      0:00 less /etc/hosts

18410 pts/4    S      0:00 /bin/bash

18432 pts/4    S+     0:00 top

This tells us that we have two shells running on that terminal. The first is looking at the hosts file while the second is running the top command. So the system does actually see that we are playing around within the second shell, its just that sudo can not log it.

Some other root level commands you can try that will not harm anything:

iptables -L

ifconfig

lsof -i

When you are convinced you have full root access, go back and check the log to verify that sudo has not logged anything:

tail /var/log/secure

When you are done with the shell in the first terminal window, type in:

exit

q

What have we learned?

We’ve learned that we cannot permit users to have full sudo access to the system and actually maintain an audit trail of what they do. Access has to be restricted to tools that do not support invoking a shell. If we are unsure, its a simple matter of checking the man pages. Try this:

man less

/

shell

The command “man less” brings up the manual for using the “less” command. Pressing the backslash key brings up the search function. When we type in the word “shell” and press the enter key, man jumps to the first instance of the string “shell” and highlights every instance. Note the section man jumped to discribes how less can invoke a shell. Now we know we cannot permit sudo access to the command “less”. A better option may be the command “cat” which does not support invoking a shell.

Another possibility is to simply use a different tool. For example the tool op was specifically designed to address some of the security shortcomings in sudo, including the ability to control shell access.

Exec Summary

Sudo is a great security tool provided you use it wisely. One of the most common mistakes made with sudo is using it to permit access to all root level tools. Sudo can generate an effective audit trail, but only if it is limited to tools which do not have shell capability.

No comments »

Posted in 2-alert, Linux Security

Tags:

Setting Up A Security Information Management System-Part5

August 14th, 2009

In my last post I discussed how a logging server uses a message’s priority value to sort incoming log messages. In this installment I’ll talk about testing connectivity, as well as how to get various gear on the wire to submit their log entries to a centralized server.

Requirements

In order for a system to submit log entries, it has to have support for Syslog. Log entries need to be transmitted in clear text to port UDP/514 on the logging server. If you are using rsyslog on the server, TCP/514 is acceptable as well.

Submitted long entries should include the following info:

  • A priority value (in <PRI> format) at the beginning of the payload
  • The name of the application submitting the log entry
  • The process ID used by the application
  • The body of the log messages

But to be honest, Syslog is not very fussy. It will attempt to record anything sent to its listening port as a long entry. If a priority value is not present, it will record the entry to whatever file is used for facility 1. Typically this is /var/log/messages.

Testing Connectivity

When deploying a new setup, I like to verify connectivity between the first few clients and the logging server. If logging is not working, you will want to be able to isolate the problem area. Typical problems include:

  • Client incorrectly configured
  • Firewall in the way (on the client, the wire, or the logging server)
  • Server incorrectly configured

You will want to monitor the messages file on the logging server to ensure the test log entry is received. On the logging server, run the following command:

tail -f /var/log/messages

Tail will open the log file read only and print the last five lines. As new log entries are received, they will get printed to the screen as well.

To generate a test log entry, I like to use Netcat. Netcat can be used from any Windows, Linux or UNIX system. From the test system, run the command:

echo ‘this is a UDP test log entry’ | nc -u -w 1 <IP address of logging server> 514

You should see the echoed portion of the command show up in the /var/log/messages file on the logging server. If not, launch a packet sniffer and see if you can determine where the failure is occurring. If you wish to test TCP connectivity as well, simply run the command again leaving out the Netcat “-u” switch:

echo ‘this is a TCP test log entry’ | nc -w 1 <IP address of logging server> 514

If both entries are received, we are ready to start pointing devices at our logging server.

Network hardware

Most network hardware supports Syslog via UDP/514. It is just a matter of going through the documentation and determining the proper command set for sending log entries to a remote server.

If you are using Cisco IOS, run the following from global configuration mode:

logging <IP address of logging server>

If you wish to change the logging facility from “local use 7” to something else, the command is:

logging facility <facility short name>

So to change logging facility to “local use 3”, the command would be:

logging facility local3

Linux and UNIX systems

There are a number of Syslog alternatives available for Linux and UNIX. In this section I’ll cover how to get Syslog and rsyslog to forward their log messages to a remote server.

Syslog

If the client is running Syslog, you will need to edit the /etc/syslog.conf file. Add the following line to the bottom of the file:

*.*                   @<IP address of logging server>

So an example would be:

*.*                   @192.168.1.150

Note the white space between the wild card match and the remote IP address MUST BE TAB CHARACTERS. If you use spaces, Sysylog will not be able to parse the file. Save and exit the file, then restart Syslog to activate the changes.

rsyslog

With rsyslog we have the option of forwarding our log messages via UDP or TCP. In either case we will need to edit the /etc/rsyslog.conf file. To forward long entries using UDP, add the following line to the end of the file:

*.* @<IP address of logging server>:<port>

So an example would be:

*.* @192.168.1.150:514

If we wish to use TCP instead, we simply use two “@” symbols:

*.* @@192.168.1.150:514

Once complete save and exit the file. You will need to restart rsyslog to activate your changes.

Windows systems

As mentioned in a previous post, Windows does not include support for Syslog. This means you will need third party software to convert your logs in real time and submit them to a logging server. The Loganalysis Web site has a list of possible solutions.

For the purposes of this post, I’ll cover Snare. It is free for use, can be commercially licensed, and have a very simple deployment process.

Once you download the software, you need to configure it for the system. This is shown in Figure #1. Snare needs to know the location of the logging server as well as what facility and severity level to use. Once complete, click the “Latest Events” menu potion to see which specific Event Viewer log entries Snare is forwarding to the logging server.

snare-config

Exec Summary

In this post I discussed testing connectivity to a logging server, as well as how to configure clients to centralize their logs. In the next post I’ll talk about what to look for in a daily reporting tool, as well as real time alerting.

No comments »

Posted in 3-err, Logging

Tags:

How To Use Windows Auto-Complete And Command Line History

August 13th, 2009

Ran into this issue at a client site today so I thought I would post some info on it. I see a lot of Windows administrators that don’t know how to fully leverage the Windows auto-complete functionality as well as the command line history. I also see Linux and UNIX administrators who get confused because these features work differently under Windows. Thought I would write up a quick how-to.

Windows auto-complete

Auto-complete permits you to type in a portion of a file name and have the system fill in the rest of the name for you. It’s a great way to save a few keystrokes. Simply type in the first couple of letters for the file name and press the <tab> key. The first file in the current directory, which starts with the letters you typed, will be filled in on the command line. This is shown in Figure #1.

win-auto-complete

For the folks that have used auto-complete on UNIX or Linux, here is where they get confused. Both platforms require you to type in enough of the command to be unique. You can then press <tab> to fill in the rest of the file. Windows works a little different.

Note in Figure #1 Windows filled in the first file name match to the character string “nets” the first time the <tab> key was pressed. If this is not the file you wanted, simply press the <tab> key a second time. You can continue to press the <tab> key and scroll through all of the possible file options.

Eventually the system will wrap and return to the first file it presented. For example pressing <tab> six times in the above example will return you to the file named “netsetup.cpl”. If you need to scroll backwards through the file options, simply hold down the <shift> key while you press <tab>.

Windows command history

Most folks realize you can press the up and down arrow keys to scroll through your command line history. Have you tried the <F7> key? If you are working in a command prompt and press the function 7 key ( <F7> ), a menu appears showing all the commands that you typed. This is shown in Figure #2. You can then use the up and down arrows, as well as the <Page Up> and <Page Down> keys, to navigate this list of commands.

prompt-history

Here’s another area where Windows works a little differently than Linux and UNIX. With Linux and UNIX, pressing the up arrow will always retrieve the last command you typed. If you have 10 sequential commands you wish to repeat, you will have to repeatedly hit the up arrow key 10 times to retrieve each command in sequence. Windows makes this process a little easier.

Try this on your Windows system:

  • Open a command prompt
  • Type the number 1 and press <enter>
  • Ignore the command not found error
  • Repeat the last two steps for numbers 2 – 10
  • Press the up arrow till you retrieve the number 5
  • Press enter
  • Press <F7>

An example is shown in Figure #3. Note the menu bar is over the number 5. This is the current reference point in the command history. While pressing the up arrow on a Linux or UNIX command line will always recalls the last command you typed, Windows shift the reference point to last recalled command. So pressing up arrow recalls the command prior to the last recalled command, not the last executed command like Linux and UNIX.

win-auto-complete2

To test this, press the <Esc> key. This will return you to the command prompt. Now press the up arrow. This will recall “4”, the command prior to “5” in the history, not “10” which was the last command executed prior to recalling “5”. Now press the down arrow until the “7” command is displayed and press <Enter>. If you now press <F7> you will notice the reference pointer has been moved to “7”, the last recalled command.

Exec Summary

Some functions operated differently at the Windows command line than their UNIX or Linux counterparts. Auto-complete and command line history are two great examples. Which implementation is preferable, unusually depends on which operating system you are most comfortable using.

No comments »

Posted in 4-notice, Windows Security

Tags: