Chapter 12

Let us have a look at how tcpdump can assist in these situations.

Situation A: No Communication on Port 500

Use tcpdump on sg1 and on sg2 at the same time:

# tcpdump -i eth1 -n -p udp port 500 or udp port 4500

The following table explains the options used here.

-i eth1	Only look on this interface. Some distributions include support for dumping on all
	interfaces. Avoid using that, as it will confuse.
-n	Do not perform reverse DNS resolution.
-p	Do not use promiscuous mode. We only want to see packets for this host.
udp port 500	Only look for UDP packets with a source or destination port of 500.
or udp port 4500	Or look for UDP packets with a source or destination port of 4500. This happens with
	NAT-Traversal.

Now repeat the --replace and --up commands mentioned before the previous two tables. If there are no problems, then each packet leaving sg1 should be seen arriving at sg2. If one of these gateways has many tunnels, you will want to limit tcpdump to only show the packets for this particular sg1-sg2 combination. You can do this by extending the tcpdump filter we used above:

sg1# tcpdump -i eth1 -n -p ip host 1.2.3.4 and '(' udp port 500 or udp port 4500 ')' sg2# tcpdump -i eth1 -n -p ip host 5.6.7.8 and '(' udp port 500 or udp port 4500 ')'

1.2.3.4 is the IP of sg2 and 5.6.7.8 is the IP of sg1. Note each system filters for the IP of the opposite system.

You may be tempted to just look at all traffic from the opposite security gateway. This can be done, but be careful if you are SSHed into one security gateway from another. You will dump port 22 traffic, which is your SSH session with your tcpdump running. Since capturing these packets shows up in tcpdump, this will be transferred to you by SSH, using port 22. This will create more port-22 traffic for tcpdump, leading to an endless loop. You can, however, also listen to everything and just exclude the port-22 traffic using:

sg1# tcpdump -i eth1 -n -p ip host 1.2.3.4 and not port 22

What we are looking for are packets that start at sg1, and do not get to sg2. It may be helpful to include the -v option to tcpdump, which will decode the initial main mode proposal, since those first two exchanges are not encrypted.

There are two possible outcomes:

A1: All Packets Arrive on sg2

If all the packets arrive on sg2, but the Pluto running on it does not acknowledge them, then the problem is likely firewalling on sg2. You can confirm this by enabling plutodebug=all, and checking that Pluto does not receive a single packet. Or you can temporarily add an ACCEPT rule for all packets from and to sg1 using:

291

Debugging and Troubleshooting

sg2: # iptables -I INPUT -s ip_of_sg1 -j ACCEPT sg2: # iptables -I OUTPUT -d ip_of_sg1 -j ACCEPT

A2: No Packets Arrive on sg2

This most likely means there is a firewall somewhere in between that is filtering your packets, or there is a NAT involved that you do not know about.

Situation B: Failure at Third Exchange

First, examine the logs on sg2. If it is complaining about being unable to find the appropriate keys, then the problem is not a communication failure. It will log that it received MI3 and complain about a failure to authenticate when RSA keys are used. If this conn uses PSK (authby=secret) then this error will appear as a failure to decrypt properly.

Assuming that there are no log entries on sg2, then the third packet may not be received. There are a number of possible reasons for this. Use tcpdump on both ends again, but include -v:

sg1# tcpdump -i eth1 -v -n -p udp port 500 or udp port 4500 sg2# tcpdump -i eth1 -v -n -p udp port 500 or udp port 4500

Look for the third exchange; it will be marked as [E].

The first exchange will look like:

11:14:25.516187 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 17, length: 320) 205.150.200.247.500 > 205.150.200.252.500: isakmp 1.0 msgid : phase 1 I ident: [|sa]

11:14:25.537388 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 17, length: 128) 205.150.200.252.500 > 205.150.200.247.500: isakmp 1.0 msgid : phase 1 R ident: [|sa]

And the second exchange will look like:

11:14:25.547023 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 17, length: 272) 205.150.200.247.500 > 205.150.200.252.500: isakmp 1.0 msgid : phase 1 I ident: [|ke]

11:14:25.772504 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 17, length: 272) 205.150.200.252.500 > 205.150.200.247.500: isakmp 1.0 msgid : phase 1 R ident: [|ke]

And the third exchange will look like:

11:14:25.781501 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 17, length: 232) 205.150.200.247.500 > 205.150.200.252.500: isakmp 1.0 msgid : phase 1 I ident[E]: [encrypted id]

11:14:25.865700 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 17, length: 360) 205.150.200.252.500 > 205.150.200.247.500: isakmp 1.0 msgid : phase 1 R ident[E]: [encrypted id]

The above is what it should look like: you should see the same thing at each end. Variations that one might see are:

•the packet may have been fragmented

•traffic on port 4500

•the IP addresses may also have changed if a NAT is involved

•any combination of the above

292

Chapter 12

Fragmentation Problem

If certificates are involved, and they are being sent inline, that may lead to I3/R3 packets that are larger than 1500 bytes, which requires that the packet be fragmented. This will be indicated by having a non-zero 'id' field, and the flags will include '[+]'. The above filter will not show the fragments.

If you are seeing fragmentation, then adjust the filter to show all packets going to the other end. Be careful, as this may result in a lot of traffic:

sg1# tcpdump -i eth1 -v -n -p ip host 1.2.3.4 and not port 22 sg2# tcpdump -i eth1 -v -n -p ip host 5.6.7.8 and not port 22

Very carefully check for fragments leaving one system and not arriving at the other system. Note that Linux sends the fragments before the initial fragment.

It is also possible that the local system is filtering the fragments itself; in which case, no packet will emerge at all. This can be due to local firewalling, but can also be due to UDP on the 2.6 kernel having the Don't Fragment bit set.

If you enable partial logging for Pluto using plutodebug=emitting, the logs will show you how big the UDP packet is that is being sent. This way you can confirm that large packets are being sent, and that receiving these packets is the problem.

The most common situation is ISPs, poorly designed routers, or over-zealous firewall admins who have filtered out fragments. Often they will claim that they have not done that. Test the situation with:

sg1# ping -s 5000 1.2.3.4

You may also want to try this using the hping2 utility:

sg1# hping2 -2 -x -y --destport 500 1.2.3.4

A way to determine that this is in fact the problem is to omit the certificate payload by putting leftsendcert=never. Copy the certificate to sg2, and point the conn at it. While you may not want to operate like this permanently, it can help to diagnose the problem.

Port 4500 is Closed

If NAT was detected, which will be logged by Pluto when you use the --up command, and you see the I3 packet leaving sg1 to port 4500, but not arriving at sg2, then the likely reason is that port 4500 has been blocked. If you see the packet arrive, but Pluto on sg2 never sees it, then the problem is probably firewalling on sg2 or on a firewall in front of sg2, or the fault of sg2's ISP.

Be careful with firewall rules: port 4500 traffic, unlike port 500 traffic, does not always originate on port 4500. For example:

# iptables -I INPUT -s 0.0.0.0/0 -d 0.0.0.0/0 -p udp --sport 500 --dport 500 -j ACCEPT

This rule will work fine for plain IKE traffic when there is no NAT involved, but you cannot use a similar rule for UDP 4500, since the source port might be a random high port instead of 4500. Instead, you need rules like:

#iptables -A INPUT -s 0.0.0.0/0 -d 0.0.0.0/0 -p udp --sport 4500 -j ACCEPT

#iptables -A INPUT -s 0.0.0.0/0 -d 0.0.0.0/0 -p udp --dport 4500 -j ACCEPT

293

Debugging and Troubleshooting

However, this is a rather liberal rule set. For example, it would permit traffic from port 4500 to port 138, which is not the intended behavior. Either place these firewall rules after all your other firewall rules (and use -A to append, not -I to insert, the rule), or further tighten the IP address of the machines you are willing to talk IKE with. Of course if you have roadwarriors, then you will need to open up IKE to the world.

Situation C: QUICK Mode Initiates, but Never Completes

This situation can be diagnosed from the logs on sg2. tcpdump does not help at all in this case.

Situation D: All IKE Messages Occur, but no Traffic Flows

If you are using KLIPS, you have access to the ipsecX interfaces. You can run tcpdump on them to see if encrypted packets on the Ethernet device are being decrypted properly and sent out on the ipsecX device. First start tcpdump again on both machines, but now on the ipsecX device:

sg1# tcpdump -i ipsec0 -n -p sg2# tcpdump -i ipsec0 -n -p

Then, send some traffic. Do you see the traffic you are sending? If sg1 is a very busy IPsec machine, then you might want to add some additional filters to limit the decrypted traffic you want to see.

If you see traffic, but the traffic does not get through to its final destination, then you could have a firewall problem on sg2, or perhaps a routing problem. Alternatively, you might not have enabled

IP forwarding in /etc/sysctl.conf.

If you see no traffic on sg1, then you may have a firewall problem there. Also read the logs on both ends. Was the SA set up properly? Look for errors from the ip or route commands.

If you see traffic on sg1, but none on sg2, then you need to investigate if the session layer ESP traffic is getting through.

Third, is there NAT involved? It could be that a NAT device will try to be helpful. As a result, the two gateways may not detect that there is a NAT involved, and may not switch from using port 500 and protocol 50 (ESP) packets to UDP port 4500 packets with encapsulated ESP as payload.

As of version 2.3.0, Pluto will log what it is going to do:

004 "sg1--sg2-net" #797: STATE_QUICK_I2: sent QI2, IPsec SA established {ESP=>0xaa6fa19a <0xa2f3b68d xfrm=3DES_0-HMAC_SHA1 NATD=none DPD=enabled}

If it says "NATD=none", and you think that there is NAT, then you may have to set

forceencaps=yes in the conn definition.

If this is not the case, then on each end issue the following:

sg1# tcpdump -i eth1 -v -n -p ip host 1.2.3.4 and ip proto 50 sg2# tcpdump -i eth1 -v -n -p ip host 5.6.7.8 and ip proto 50

You should see ESP packets with SPI 0xaa6fa19a leaving sg1 (this the => in the log entry), and ESP packets with SPI 0xa2f3b68d arriving on sg1. On sg2, you should see the opposite.

Ensure that you do not only see traffic going in one direction.

294