Practical Overview of the IPsec Protocol

Crypto Requirements for IPsec

In conclusion, we need a strong cryptographic checksum to protect the integrity of our packets using a symmetric cipher. This cipher will be keyed by a session key, which we need to exchange after we have achieved privacy using a DH key exchange, and have authenticated each other based on trusted public key encryption.

We want to add a digital signature to as much of the packet as we can. That is, we want a digital signature that covers all the IP headers that don't change plus the data (payload) of the packet. Since there is no space in the original TCP or UDP headers, we will have to define a new IP protocol. And it turns out we don't have just one, but two new IP protocols: Authenticated Header (AH) and Encapsulated Security Payload (ESP).

We also learned that we need a clever system to exchange session keys based on public key systems. The protocol in the IPsec suite that is used for that is called the Internet Key Exchange (IKE).

These protocols are explained in depth later in this chapter.

IPsec: A Suite of Protocols

There is not a single IPsec protocol. IPsec is in fact a collection of standards (and drafts, because the IETF process is very slow) that all deal with using cryptography to ensure authenticity and in almost all cases to also guarantee confidentiality of the content of the IP packets. Most of the standards documents contain details of the cryptographic ciphers and algorithms used. The intent of this chapter is to cover what you need to know from a practical point of view, without going into all the details and design decisions.

The IPsec protocols can be split into two main categories: packet handling and trust relationship management. Packet handling is usually done by the operating system kernel itself, since it requires speed, efficiency, and low latency that are easier to offer at the low-level processing of the kernel.

The trust relationship management is not as time sensitive, since it only happens at the start and at the refresh intervals of an IPsec connection, which is usually about once an hour. It is also a very complex process, requiring a lot of very complex code that is much better done outside the kernel, as a regular program running on the computer.

Kernel Mode: Packet Handling

The kernel deals with the individual packets sent and received by the computer. This is sometimes called the forwarding plane. It involves turning normal IP packets into secure IPsec packets, carrying out encryption, decryption, signing, encapsulation, and decapsulation of the packets. These techniques all involve changing and verifying packets, and are normally performed by the kernel.

32

Chapter 2

Authentication Header (AH)

Authentication Header (RFC 2404) is the first new network protocol that was introduced. It received IP protocol number 51. (Other examples are the TCP protocol, IP protocol 6, and the UDP protocol, IP protocol 17. On a computer running Linux, you can find a list of IP protocols in /etc/protocols). Don't be misled by the name: AH does not just authenticate the header of an IP packet, but authenticates the data (payload) as well as parts of the header.

When two machines are configured with secret keys to communicate using the AH protocol, they agree on a unique number identifying the connection. This number is called the Security Parameter Index (SPI).

The machines then set the Sequence Number (SN) to 0. For each packet that is to be sent, the SN is increased by one to prevent replay attacks. In a replay attack, a malicious listener intercepts an encrypted packet, and without being able to read that captured data, resends it over and over again. Without the SN, the two computers that are communicating with encryption would not be able to distinguish a genuine packet from a copy of a previous packet, since the cryptography checks will be satisfied by the replayed packet. After all, it was a valid packet. The SN prevents these attacks, because now the packet can be correctly identified as an old retransmitted packet, and discarded.

A cryptographic checksum is then calculated for the packet, using a hashing algorithm, usually MD5 or SHA1, using a secret key that only the sender and receiver know. This cryptographic checksum is called the integrity check value (ICV). The SPI, SN, and ICV form the important part of the AH header. The SN and ICV together with the hash are also called the HMAC (for keyed Hash Message Authentication Code). The term HMAC actually refers to the specific method used to calculate the ICV, but colloquially the result is sometimes called the HMAC.

The rest of the header consists of various administration pointers for these header parameters within the packet. This packet is then sent to the other IPsec endpoint. Upon receiving the packet, it also calculates the ICV for the packet using its own copy of the secret key. If the calculated ICV matches the ICV stored in the received packet, the packet is authentic. The IPsec header of the packet is then removed, leaving a regular IP packet which can then continue on into the kernel like any other unencrypted packet. If the ICV does not match, for instance because a NAT device rewrote the IP address, or some host on the way changed any data in the packet, the packet is discarded by the kernel before any application gets to see the malicious or broken packet.

AH only provides authentication and does not encrypt the payload. Anyone who can sniff the AH packets can see who is communicating with whom, and what they are saying. However attackers cannot change or inject any packets, since they cannot forge the ICV in the packets since they do not have access to the secret key used to compute the ICV that assures packet integrity.

The source and destination address are part of the data protected by AH, so they cannot be spoofed. But neither can they be rewritten by a NAT device without causing the packet to be discarded as invalid, because the rewriting does not match the computer AH HMAC. Therefore, AH does not work together with NAT. This can be seen as a feature or as a bug, depending upon your threat model.

Since AH on its own does not offer encryption, it is hardly used at all.

33

Practical Overview of the IPsec Protocol

Encapsulated Security Payload (ESP)

ESP is a much more useful new protocol of the IPsec suite. It received protocol number 50. Its job is not only to authenticate the packet, like AH, but also to add a security policy to the packet, and optionally encrypt it. So it has some of the same properties as AH, such as the SPI, SN, and ICV. For the decryption of the packet content, both ends need to have the secret key with which the packet has been encrypted, and can be decrypted. You can use ESP without encryption by using the NULL encryption. The protocol header is different to accommodate the additional encryption settings and some padding, which is necessary due to the way some ciphers work.

Originally ESP provided no authentication, so if you wanted integrity and privacy, you would build an IP packet in ESP in AH. ESP now provides for authentication as well, in a very similar form to AH. So now you either use ESP with authentication or you have an AH header around the ESP header. In practice however, the ESP implementations do not allow ESP without authentication, so the ESP in AH construct of packets is no longer used.

The only reason AH is separate from ESP is because of the US export restrictions that were in effect when they were devised.

Transport and Tunnel Mode

There are two modes for IPsec connections. One is called transport mode, the other is called tunnel mode. Transport mode is only used for communication between two hosts. There is only one IP header, and the protocol header within the IP header (such as TCP or UDP) is protected. In transport mode, the goal is to safely transport the packet itself. In other words, the packet itself is the payload. In tunnel mode, there is a clear distinction between the transport packet and the payload, which is a complete packet in its own right. An entire IP packet, with its own full IP header, forms the payload of a newly created IPsec packet. On top of that, there is an additional IPsec policy that dictates what kinds of packets are allowed to travel over a tunnel mode connection. You could call this a built-in ingress firewall.

You can compare this with cars. Transport mode equates to simply driving a car from one place to the other. Tunnel mode is more like driving the car onto a train, which then takes the car to its destination, where the car is unloaded again. You would need to obtain a valid train ticket for your car to be allowed to be put on the train.

The only time transport mode is still useful today is when you perform your own encapsulation, as it saves another layer of IP header, or about 20 bytes per packet. This is the case for Microsoft's LT2P protocol, which uses PPP encapsulation.

The reason for the existence of both modes is now mostly historic. Host-based stacks would like to get rid of tunnel mode, because it can be done with transport mode, but third-party IPsec stacks that put themselves between the operating system and the user, also called bump in the stack-based stacks, would like to get rid of transport mode because it can be done with tunnel mode.

34

Chapter 2

Choosing the IPsec Mode and Type

In almost all cases, you will use ESP in tunnel mode. This is what people call a Virtual Private Network (VPN). ESP allows you to encrypt your packets as well as authenticate them, and tunnel mode means that you can connect not only a single host to another single host, but also that you can hook up subnets to subnets. And you can even hide the source and destination addresses of those packets, since in tunnel mode the entire packet, including source address, destination address, and port are encrypted. Even if you are setting up a host to host connection, you should use tunnel mode, so that all the IP options of the packets can be encapsulated and encrypted. In fact, two well known people in the cryptography world, Bruce Schneier and Niels Ferguson, have argued for AH and transport mode to be got rid of altogether.

ESP tunnel mode offers all the options you could want at a cost of a very few bytes per packet at the most.

The only time we will use transport mode in this book is when we're discussing how to set up Microsoft L2TP, or Layer 2 Tunneling Protocol.

Some people might be tempted to use AH in an effort to reduce the CPU load of some IPsec connection, for example if they believe that their network is already encrypted safely. People with WiFi networks who encrypt using the (very weak) WEP encryption regularly try to disable IPsec encryption because the CPU on most WiFi equipment is not very powerful. We feel we must stress the point that most of these vendor-made encryption protocols are broken sooner rather than later. And even if they are not broken yet, those protocols have not withstood the years of public scrutiny and research that the IPsec protocol has had. Don't be too clever. Do not try to be too efficient. Stick to ESP with tunnel mode, unless you are trying to do L2TP interop with Microsoft.

Do not use Authenticated Header (AH) or Transport mode. Only use Encapsulated Security Payload (ESP) in Tunnel mode. This is the most secure solution from a cryptographic point of view.

The Kernel State

The kernel needs to manage the IPsec connections it has been made aware of, and it does this mainly through Security Policies (SP). These determine which packets need or do not need processing. For example, one important rule is: Never accept unencrypted traffic from a host for which we have an established, working IPsec connection. Another rule could be: If we do not have an established IPsec connection for this host, an attempt should be started to establish one before allowing this packet through. Security Policies are stored in the Security Policy Database (SPD).

Furthermore the kernel has to keep track of all IPsec connections and their corresponding parameters, such as SPI, mode, cryptographic keys, and SPs that apply to it. All this information for a single IPsec connection is called an IPsec Security Association (IPsec SA). This is a unidirectional concept. Two are required for packets to flow in both directions.

35