Chapter 2

Final Note on Protocols and Ports

A common mistake is that people who are configuring firewalls think that AH and ESP are port numbers. They then write ACCEPT rules for ports 50 and 51 and think this will cause IPsec packets to be allowed to pass. But this is wrong. AH and ESP are IP protocols, just like TCP, UDP and ICMP are IP protocols. They are not ports. And like ICMP, the AH and ESP protocols do not themselves have ports.

Usermode: Handling the Trust Relationships

The other aspect is the creation of a trust relationship between hosts on the Internet. This is also called the control plane. It involves the creation of a secure communication channel and the exchange of cryptographic keys. It also involves many choices and options and a lot of state information. This part is usually implemented in a process that runs on the OS continuously listening for requests for new IPsec connections. Programs that listen for incoming connections are typically called daemons.

These two layers—the userland daemon and the kernel IPsec stack—talk through a special socket interface, usually the PF_KEY interface. Most of the RFCs in the IPsec protocol suite are not actually about packet handling, but about these trust relationships. Handling packets is a relatively straightforward process. A packet comes in, matches a ruleset, is transformed into an IPsec packet, and is sent on.

But handling a whole range of different kinds of trust relationships becomes very complex very quickly. This is one of the reasons why manual keying is so discouraged. Keys need to be destroyed and renewed regularly, and this task is not something a system administrator should do every hour. Instead, a trust relationship is established by other means, and session keys are agreed on and loaded into the kernel by a special daemon. This daemon will talk using the Internet Key Exchange (IKE) protocol.

The IKE Protocol

The IKE protocol is a very complex protocol, and involves many difficult cryptographic operations, and requires even more options to be exchanged between two hosts who want to communicate via IPsec. IKE is therefore always implemented as a userland process. IKE operations can be roughly split in two parts, or phases.

Phase 1: Creating the ISAKMP SA

Phase 1 deals with obtaining privacy through a Diffie-Hellman key exchange, which ensures there is no eavesdropper. Next, the two hosts perform an out-of-bound verification of the other based on the type and content of the ID they receive, to prevent a man-in-the-middle attack. The type of ID can vary, but it will be based on IP address, hostname, email address, or ASN.1 Distinguished Name (DN).

37

Practical Overview of the IPsec Protocol

ID Type	Typical Example
ID_IPV4	193.110.157.77
ID_FQDN	lists.openswan.org
ID_USER_FQDN	[e-mail]
DER_ASN1_DN	C=CA,L=Toronto,O=Xelerance ,CN=lists.openswan.org, E=[e-mail]

The out-of-bound verification can happen in various ways. One can use a PreShared Key (PSK), but much better is to use public key cryptography, such as RSA, to prove one's identity. These raw RSA keys can come from a PGP-signed email. Or the keys can come from DNS, preferably protected by DNSSEC to prevent attacks on the DNS, or they can come from X.509 certificates signed by a mutually trusted third party, a so-called Certificate Agency (CA). If one person is setting up both ends, it is of course possible to use Secure Shell (ssh or scp) to set up raw public RSA keys on both machines.

What is important to realize is that phase 1 needs to convey the identity of the remote host, and the host that receives the supposed identity from a remote host should have some way of verifying that claim. If phase 1 is successful, the two hosts have established an Internet Security Association Key Management Protocol Security Association (ISAKMP SA). This is often called a Phase 1 SA.

This exchange will also state whether the identification is based on PSK or RSA public keys. Because this information is exchanged very early on in the negotiation, these decisions need to be made very quickly. This actually hampers certain situations where it is not clear what type of exchange is requested in the first few packets. One such difficult situation is a client using PSK on a dynamic IP address, which is not recommended for precisely this reason.

The ISAKMP SA also needs to be re-negotiated at regular times to prevent overusing a single cryptographic session key.

Main Mode

Establishing a successful phase 1 connection requires a few packets to be sent and received before the ISAKMP SA is established. This normal method is called Main Mode. However it has the longest latency, because it involves sending and receiving several packets one after the other. These packets set up a full Diffie-Hellman key exchange, before sending any information about the requested IPsec connection. Because of these extra packets, it is both the safest mode of operation, and the most flexible, since both parties can safely and privately request options, and when the other end denies a certain request, a less favorable but still acceptable alternative option can be tried.

Aggressive Mode

Some vendors wanted a faster mode, where less packets would be needed, so that the overall latency to establish a phase 1 ISAKMP SA would be shorter. This mode is called Aggressive Mode, and to reduce the number of packets needed, this mode requires that some CPU-intensive

38

Chapter 2

tasks involving the Diffie-Hellman key exchange are already done before the first packets are sent and received. The problem is that this leads to an easy denial-of-service attack: just sending bogus initial Aggressive Mode packets to a host can bring it to a grinding halt.

Main Mode versus Aggressive Mode

A bigger vulnerability in Aggressive Mode is that to reduce the number of packets sent, the hash of the PreShared Key is transmitted before encryption has been enabled. These packets can be captured, and a brute force or dictionary attack can be run against them, for instance using a program such as ikecrack. In Main Mode however, this hash is sent only after the encryption has been enabled.

With Aggressive Mode, there is also no way to negotiate the most favorable options. The initial packet has to have all the right options in the request, or the remote end will have no choice but to say "I cannot do this." When combined with XAUTH, a man-in-the-middle attack becomes possible, and even trivial when the attacker is a valid VPN client on the network.

The limitation of negotiation of options has another drawback. Imagine that at some point in the future, the one cipher you had hard-coded into all your configurations for Aggressive Mode has been broken by a clever cryptographer. You now have to reconfigure all your clients, while if Main Mode had been used, all you would have to do would be to disallow that one broken cipher on the gateway, and the Main Mode negotiation would just pick another of the ciphers that it supports.

Avoid using Aggressive Mode since it is known to be unsafe in certain deployments, and restricts the flexibility of your configurations.

Phase 2: Quick Mode

When the ISAKMP SA is established, 'Phase 2' can start. This is also called 'Quick Mode'. Phase 2 involves creating key material for the 'IPsec SA'. That is, the key material that the program needs to communicate to the kernel for use in the IPsec SA.

Phase 2 also involves agreeing many cryptographic parameters, such as which ciphers and algorithms to use, and how long a session key may be used for, but also further information, such as for which source and destination addresses the IPsec SA will be valid. Of course it also contains which 'transform' to use, that is, which kind of protocol to use (AH or ESP) and which mode (Transport or Tunnel mode). The IPsec SA information is communicated from the userland process to the kernel using a PFKEY socket. This IPsec SA is then stored in the SAD.

The term Quick Mode is ambiguous. Some refer to a Phase 2 SA to mean the IPsec SA, but throughout the book we will use the term IPsec SA for this purpose.

Perfect Forward Secrecy (PFS)

Another option that is strongly recommended, but not available on all IPsec products, is Perfect Forward Secrecy (PFS). This is also negotiated and must be agreed upon by both ends for phase 2 to complete successfully.

39

Practical Overview of the IPsec Protocol

PFS ensures that even if your current private key is compromised, all past communication that has been sniffed and stored cannot be decrypted with this private key. It works by using a session key that is discarded after use.

VendorIDs

To make it easier to work around bugs in certain IPsec implementations, one option introduced is VendorID. It identifies the vendor, device, and sometimes even the firmware version. VendorIDs can assist in debugging a certain IPsec connection, but of course they can also pose a security risk. You might not want to announce a VendorID of a known vulnerable device for instance. If you are accepting IPsec negotiations from dynamic IPs (roadwarriors), you might have a rogue client attempting to connect, and you would give out the VendorID before you have determined this client cannot authenticate with you and should be denied. By then it might already be too late.

Multiple VendorIDs can be sent. Usually VendorIDs are strings which are hashed using MD5, which has a few advantages. First, you have to know the original string to understand the MD5 hash, so it is harder for information to leak. Second, it makes all the VendorIDs equally long in size (96 bits), which greatly simplifies the programming effort, as 96 bits happens to be the length of the md5sum output. Every parameter space in IKE has private-use values and in order to avoid conflicts among vendor-proprietary extensions, the VendorID is used to qualify whose private use it is. It has also turned out to be useful for public extensions, options that span multiple or all vendors. For example, the capability to do ESPinUDP encapsulation, to break through NAT (called IPsec NAT-Traversal), is advertised using a VendorID.

Some vendors send proprietary extensions and information through additional VendorIDs, most notably Nortel, Cisco, and Microsoft. Since we often only know the MD5 hashes, we have no idea what some of these VendorIDs actually signify.

Dead Peer Detection (DPD)

DPD is an addition to the IKE protocol, also using VendorIDs to see if the remote IPsec gateway is still up. This is done by sending and responding to probe packets over the ISAKMP SA. It works much like the ping command, but is not supported by all vendors, being notably absent from Microsoft's built-in IPsec client. This is somewhat unfortunate, since DPD would be most useful on roaming end-user laptops.

ModeConfig

ModeConfig is an extension to the IKE protocol for interactiveness. One peer can ask the other peer for something and get an answer back. This can be a username, a password, an IP address, an IP address assignment (via DHCP), policies of the tunnel, or cryptographic token responses such as SecureID.

XAUTH

XAUTH is a custom authentication extension to the IKE protocol to work around various operational issues when deploying IKE. It is actually implemented with ModeConfig and VendorIDs, and provides for an additional user and password identification. This user/password scheme can include onetime passwords, SecureID tokens, or hooks to other authentication mechanisms such as Radius, PAM, or LDAP.

40