Understand IPsec IKEv1 Protocol

Contents
Introduction
Prerequisites
Requirements
Components Used
IPsec
IKE Protocol
IKE Phases
IKE Modes (Phase 1)
Main Mode
Aggressive Mode
IPsec Mode (Phase 2)
Quick Mode
IKE Glossary
Main Mode Packet Exchange
Main Mode 1 (MM1)
Identify Two Simultaneous Negotiations
Main Mode 2 (MM2)
Main Mode 3 and 4 (MM3-MM4)
Main Mode 5 and 6 (MM5-MM6)
Quick Mode (QM1, QM2, and QM3)
Aggressive Mode Packet Exchange
Main Mode vs Aggressive Mode
IKEv2 vs IKEv1 Packet Exchange
Policy-Based vs Route-based
Policy-Based VPN
Route-Based VPN
Common Issues for Traffic Does Not Receive through the VPN
ISP Blocks UDP 500/4500
ISP Blocks ESP
Related Information

Introduction
This document describes the Internet Key Exchange (IKEv1) protocol process for a Virtual Private
Network (VPN) establishment in order to understand the packet exchange for simpler troubleshoot
for any kind of Internet Protocol Security (IPsec) issue with IKEv1.

Contributed by Amanda Nava, Cisco TAC Engineer.

Prerequisites
Requirements

Cisco recommends that you have knowledge of basic security concepts:

● Authentication
● Confidentiality
● Integrity
● IPsec

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of
the devices used in this document started with a cleared (default) configuration. If your network is
live, ensure that you understand the potential impact of any command

IPsec
IPsec is a suite of protocols that provides security to Internet communications at the IP layer. The
most common current use of IPsec is to provide a Virtual Private Network (VPN), either between
two locations (gateway-to-gateway) or between a remote user and an enterprise network (host-to-
gateway).

IKE Protocol
IPsec uses the IKE protocol to negotiate and establish secured site-to-site or remote access virtual
private network (VPN) tunnels. IKE protocol is also called the Internet Security Association and
Key Management Protocol (ISAKMP) (Only in Cisco).

There are two versions of IKE:

● IKEv1: Defined in RFC 2409, The Internet Key Exchange

● IKE version 2 (IKEv2): Defined in RFC 4306, Internet Key Exchange (IKEv2) Protocol

IKE Phases

ISAKMP separates negotiation into two phases:

● Phase 1: The two ISAKMP peers establish a secure and authenticated tunnel, which protects
ISAKMP negotiation messages. This tunnel is known as the ISAKMP SA. There are two
modes defined by ISAKMP: Main Mode (MM) and Aggressive Mode.
● Phase 2: It negotiates key materials and algorithms for the encryption (SAs) of the data to be
transferred over the IPsec tunnel. This phase is called Quick Mode.
In order to materialize all the abstract concepts, the Phase 1 tunnel is the Parent tunnel and phase
2 is a sub tunnel, this image illustrates the two phases as tunnels.
 Note: Phase 1 (ISAKMP) Tunnel protects the Control Plante VPN traffic between the two
gateways. Control Plane traffic can be Negotiation packets, information packages, DPD,
keepalives, rekey, etc. ISAKMP negotiation uses the UDP 500 and 4500 ports to establish a
secure channel.

Note: Phase 2 (IPsec) Tunnel protects the Data Plane traffic that passes through the VPN
between the two gateways. The algorithms used to protect the data are configured in Phase
2 and are independent of those specified in Phase 1.
The protocol used to encapsulate and encrypt these packets is the Encapsulation Security
Payload (ESP).

IKE Modes (Phase 1)

Main Mode

An IKE session begins when the initiator sends a proposal or proposal to the responder. The first
exchange between nodes establishes the basic security policy; the initiator proposes the
encryption and authentication algorithms to be used. The responder chooses the appropriate
proposal (we'll assume a proposal is chosen) and sends it to the initiator. The next exchange
passes Diffie-Hellman public keys and other data. All further negotiation is encrypted within the
IKE SA. The third exchange authenticates the ISAKMP session. Once the IKE SA is established,
IPSec negotiation (Quick Mode) begins.

Aggressive Mode

Aggressive Mode squeezes the IKE SA negotiation into three packets, with all data required for
the SA passed by the initiator. The responder sends the proposal, key material, and ID, and
authenticates the session in the next packet. The initiator replies and authenticates the session.
Negotiation is quicker, and the initiator and responder ID pass in the clear.

IPsec Mode (Phase 2)

Quick Mode

IPSec negotiation, or Quick Mode, is similar to an Aggressive Mode IKE negotiation, except
negotiation, must be protected within an IKE SA. Quick Mode negotiates the SA for the data
encryption and manages the key exchange for that IPSec SA.

IKE Glossary
● A security association (SA) is the establishment of shared security attributes between two
network entities to support secure communication. An SA includes attributes such as
cryptographic algorithm and mode; traffic encryption key; and parameters for the network data
to be passed over the connection.
● The vendor IDs (VID) are processed to determine whether the peer supports the NAT-
Traversal, Dead Peer Detection feature, Fragmentation, etc.
● Nonce: a randomly generated number that the initiator sends. This nonce is hashed along
with the other items with the agreed key used and is sent back. The initiator checks the cookie
and the nonce and rejects any messages which do not have the right nonce. This helps
prevent replay since no third party can predict what the randomly generated nonce is.
● Key-exchange (KE) information for the Diffie-Hellman (DH) secure key-exchange process.
● Identity Initiator/responder (IDi/IDr) is used to send out authentication information to the
peer. This information is transmitted under the protection of the common shared secret.
● Diffie–Hellman (DH) key exchange is a method of securely cryptographic algorithms
exchange over a public channel.
● The IPSec shared key can be derived with the DH used again to ensure Perfect Forward
Secrecy (PFS) or the original DH exchange refreshed to the shared secret derived previously.

Main Mode Packet Exchange

Each ISAKMP packet contains payload information for the tunnel establishment. The IKE glossary
explains the IKE abbreviations as part of the payload content for the packet exchange on Main
Mode as shown in this image.
Main Mode 1 (MM1)

To set the terms of the ISAKMP negotiations, you create an ISAKMP policy, which includes:

● An authentication method, to ensure the identity of the peers.

● An encryption method, to protect the data and ensure privacy.
● A Hashed Message Authentication Codes (HMAC) method to ensure the identity of the
sender, and to ensure that the message has not been modified in transit.
● A Diffie-Hellman group to determine the strength of the encryption-key-determination
algorithm. The security appliance uses this algorithm to derive the encryption and hash keys.
● A limit to the time the security appliance uses an encryption key before it gets replaced.
The first packet is sent by the Initiator of the IKE negotiation as shown in the image.
 Note: The Main Mode 1 is the first packet of the IKE negotiation. Therefore, the Initiator SPI
is set to a random value while Responder SPI is set to 0. In the second packet (MM2) the
Responder SPI must be replied to with a new value and the entire negotiation maintains the
same SPIs values.

If the MM1 is captured and a Wireshark network protocol analyzer is used, the SPI value is within
the Internet Security Association and Key Management Protocol content as shown in the image.

Note: In the case, the MM1 packet gets lost in the path or there is no MM2 reply, the IKE
negotiation keeps the MM1 retransmissions until the maximum number of retransmissions is
reached. At this point, the Initiator keeps the same SPI until the next negotiation is triggered
again.

Tip: Initiator and Responder SPIs identification is very helpful to identify multiple negotiations
for the same VPN and narrow down some negotiation issues.

Identify Two Simultaneous Negotiations

On the Cisco IOS® XE platforms, the debugs can be filtered per tunnel with a conditional for the
remote IP address configured, however, the simultaneous negotiations are displayed on the logs,
and there is no way to filter them. It is needed to do it manually. As previously mentioned, the
whole negotiation keeps the same SPI values for Initiator and responder. In case a packet is
received from the same peer IP address but the SPI does not match the previous value tracked
before the negotiation reaches the maximum number of retransmission, it is another negotiation
for the same peer as shown in the image.
 Note: The example shows simultaneous negotiation for the first packet in the negotiation
(MM1), however, this can occur at whatever negotiation point. All the subsequent packets
must include a value different from 0 on responder SPI.

Main Mode 2 (MM2)

In the Main Mode 2 packet, the responder sends the selected policy for the proposals matched,
and the responder SPI is set to a random value. The entire negotiation maintains the same SPIs
values. The MM2 replies to MM1 and the SPI responder is set to a different value from 0 as shown
in the image.

If the MM2 is captured and a Wireshark network protocol analyzer is used, the Initiator SPI and
Responder SPI values are within the Internet Security Association and Key Management Protocol
content as shown in the image.
Main Mode 3 and 4 (MM3-MM4)

The MM3 and MM4 packets are still unencrypted and unauthenticated and the Secret key
exchange takes place. MM3 and MM4 are shown in the image.

Main Mode 5 and 6 (MM5-MM6)

The MM5 and MM6 packets are already encrypted but still unauthenticated. On these packets, the
authentication takes place as shown in the image.

Quick Mode (QM1, QM2, and QM3)

Quick mode occurs after the Main monde and the IKE has established the secure tunnel in phase
1. Quick Mode negotiates the shared IPSec policy, for the IPSec security algorithms and manages
the key exchange for the IPSec SA establishment. The nonces are used to generate new shared
secret key material and prevent replay attacks from bogus SAs generated.

Three packets are exchanged in this phase as shown in the image.

Aggressive Mode Packet Exchange
The Aggressive Mode squeezes the IKE SA negotiation into three packets, with all data required
for the SA passed by the initiator.

● The responder sends the proposal, key material, and ID, and authenticates the session in the
next packet.
● The initiator replies and authenticates the session.
● Negotiation is quicker, and the initiator and responder ID pass in the clear.
The image shows the payload content for the three packets exchanged on Aggressive mode.
Main Mode vs Aggressive Mode
Compared to the Main Mode, Aggressive Mode comes down to three packages::

● AM 1 absorbs MM1 and MM3

● AM 2 absorbs MM2, MM4, and part of the MM6. This is where the vulnerability of Aggressive
Mode comes from. The AM 2 makes up the IDr and Authentication unencrypted, unlike the
Main Mode this information is encrypted.
● AM 3 provides the IDi and the Authentication, those values are encrypted.

IKEv2 vs IKEv1 Packet Exchange

In the IKEv2 negotiation, fewer messages are exchanged to establish a tunnel. IKEv2 uses four
messages; IKEv1 uses either six messages (in the main mode) or three messages (in aggressive
mode).

The IKEv2 message types are defined as Request and Response pairs. The image shows the
packets comparison and payload content of IKEv2 versus IKEv1.
 Note: This document does not describe deeper the IKEv2 Packet exchange. For more
references, navigate to IKEv2 Packet Exchange and Protocol Level Debugging.

Policy-Based vs Route-based
Policy-Based VPN

As the name states, A policy-based VPN is an IPsec VPN tunnel with a policy action for the
transit traffic that meets the policy's match criteria. In the case of Cisco devices, an Access List
(ACL) is configured and attached to a crypto map to specify the traffic to be redirected to the VPN
and encrypted.

The traffic selectors are the subnets or hosts specified on the policy as shown in the image.
Route-Based VPN

A Policy is not needed and the traffic is redirected toward the tunnels with routes and It supports
dynamic routing over the tunnel interface. The traffic selectors (traffic encrypted through the VPN)
are from 0.0.0.0. to 0.0.0.0 by default as shown in the image.

Note: Due to the Traffic selectors are 0.0.0.0, any host or subnet is included within,
therefore, only one SA is created. There is an exception for Dynamic tunnel. This document
does not describe dynamic tunnels.
The Policy and Route-based VPN can be materialized as shown in the image.

Note: Unlike Route-based VPN with only one SA created, the Policy-based VPN can create
multiples SA. As an ACL is configured, each statement on the ACL (if they are different
between them) creates a sub-tunnel.

Common Issues for Traffic Does Not Receive through the

VPN
ISP Blocks UDP 500/4500

It is a very common issue that the Internet Services Provider (ISP) blocks the UDP 500/4500
ports. For an IPsec tunnel establishment, two different ISPs can be engaged and one of them can
block the ports and the other allows them.

The image shows the two scenarios where an ISP can block the UDP 500/4500 ports in only one
direction.
 Note: Port UDP 500 is used by the Internet key exchange (IKE) for the establishment of
secure VPN tunnels. UDP 4500 is used when NAT is present in one VPN endpoint.

Note: When the ISP Blocks UDP 500/4500, the IPsec tunnel establishment is affected and it
does not get up.

ISP Blocks ESP

Another very common issue on IPsec tunnels is, the ISP blocks the ESP traffic however, it allows
the UDP 500/4500 ports. An example, the UDP 500/4500 ports are allowed in bidirectional ways,
therefore, the tunnel is successfully established but the ESP packets are blocked by the ISP or
ISPs in both directions, this causes the encrypted traffic through the VPN to fail as shown in the
image.
 Note: When the ISP Blocks ESP packets, the IPsec tunnel establishment is successful but
the traffic encrypted is affected. Wich, it can be reflected with the VPN up but the traffic does
not work over it.

Tip: The scenario where the ESP traffic is blocked only in one direction can be present as
well, the symptoms are the same but it can be easily found with the tunnel statistics
information, encapsulation, decapsulation counters, or RX and TX counters.

Related Information
● KEv2 Packet Exchange and Protocol Level Debugging
● The Internet Key Exchange (IKE) - RFC 2409
● Internet Key Exchange (IKEv2) Protocol
● Technical Support & Documentation - Cisco Systems