Buddha Institute of Technology, GIDA, Gorakhpur

Cryptography & Network Security-KCS074

Unit-5
IP Security & System Security

Prepared by:
Mr. Satyam Kumar Singh
Assistant Professor
Computer Science & Engineering
Buddha Institute of Technology,
Gorakhpur
IP Security (IPsec)
• IPsec is a suite of protocols developed to secure Internet Protocol (IP) communications by
authenticating and encrypting each IP packet in a communication session.
• It operates at the network layer, providing security features like confidentiality, integrity, and
authentication.
• Below are the main components and concepts:
IPSec (IP Security) architecture uses two protocols to secure the traffic or data flow. These
protocols are ESP (Encapsulation Security Payload) and AH (Authentication Header). IPSec
Architecture includes protocols, algorithms, DOI, and Key Management. All these
components are very important in order to provide the three main services:
• Confidentiality

• Authentication

• Integrity
IPSec Architecture & Packet Format
IPsec architecture includes the following:
• Security Protocols:
• Authentication Header (AH): Ensures data integrity and authenticity.
• Encapsulating Security Payload (ESP): Provides confidentiality, along with
integrity and authentication.
• Security Associations (SA): Define the parameters for IPsec sessions,
such as algorithms, keys, and IPsec mode.
• Transport Mode: Encrypts only the payload of the IP packet.
• Tunnel Mode: Encrypts the entire IP packet, adding a new IP header.
• Key Management: Uses Internet Key Exchange (IKE) to handle keys
and establish SAs dynamically.
Authentication header
The AH protocol ensures the integrity
and authenticity of data packets:
• Functionality:
• Protects against tampering and
spoofing.
• Adds an AH header between the IP
header and the payload.
• Mechanism:
• Uses a keyed-hash algorithm (e.g.,
HMAC-SHA) for authentication.
• Does not provide encryption, so it
does not ensure confidentiality.
Encapsulating Security Payload (ESP)
ESP provides encryption to ensure data
confidentiality in addition to
authentication and integrity:
• Functionality:
• Encrypts the payload to keep it
confidential.
• Optionally supports integrity checks using
HMAC.
• Mechanism:
• Adds an ESP header, encrypted payload,
and ESP trailer.
• Can operate in transport mode or tunnel
mode.
1. Security Parameter :

• Security parameters are assigned a size of 32 bits for use

• Security Parameter is mandatory to security parameter in ESP for security links and associations

2. Sequence Number:

• The sequence number is 32 bits in size and works as an incremental counter.

• The first packet has a sequence number 1 assigned to it whenever sent through SA

3. Payload Data:

• Payload data don’t have fixed size and are variable in size to use

1. It refers to the data/ content that is provided security by the method of encryption
4. Padding:
• Padding has an assigned size of 0-255 bytes assigned to it.
• Padding is done to ensure that the payload data which needs to be
sent securely fits into the cipher block correctly, so for this padding
payloads come to the rescue.
5. Pad Length:
• Pad Length is assigned the size of 8 bits to use
• It is a measure of pad bytes that are preceding
6. Next Header:
• The next header is associated with a size of 8 bits to use
• It is responsible for determining the data type of payload by studying
the first header of the payload
7. Authentication Data:
• The size associated with authentication data is variable and never
fixed for use-case
• Authentication data is an optional field that is applicable only when
SA is selected. It serves the purpose of providing integrity
Working of ESP:
1.Encapsulating Security Payload supports both main Network
layer protocols: IPv4 and IPv6 protocols.

2.It performs the functioning of encryption in headers of Internet

Protocol or in general say, it resides and performs functions in IP
Header.

3.One important thing to note here is that the insertion of ESP is

between Internet Protocol and other protocols such as UDP/
TCP/ ICMP.
Modes in ESP
Modes in ESP:

• Encapsulating Security Payload supports two modes, i.e. Transport mode, and tunnel mode.
• Tunnel mode:
1. Mandatory in Gateway, tunnel mode holds utmost importance.

2. Here, a new IP Header is created which is used as the outer IP Header followed by ESP.

• Transport mode:
1. Here, IP Header is not protected via encryption or authentication, making it vulnerable to threats

2. Less processing is seen in this mode, so the inclusion of ESP is preferred

Below listed are the advantages of Encapsulating Security Payload:
1. Encrypting data to provide security

2. Maintaining a secure gateway for data/ message transmission

3. Properly authenticating the origin of data

4. Providing needed data integrity

5. Maintaining data confidentiality

6. Helping with antireplay service using authentication header

Below listed are the disadvantages of Encapsulating Security Payload:

1. There is a restriction on the encryption method to be used

2. For global use and implementation, weaker encryptions are mandatory to use
Combining Security Associations
• Security Associations (SAs) in IPsec define the policies and cryptographic algorithms used
to secure communications.
• A single SA applies to one direction of communication (unidirectional) and supports
either Authentication Header (AH) or Encapsulating Security Payload (ESP).
• To achieve more robust security or to meet complex security requirements, multiple SAs
can be combined.
Methods of Combining Security Associations
1.Transport Adjacency
1. Multiple SAs are applied sequentially to a single IP packet in transport mode.
2. Example:
1. One SA applies the AH protocol for integrity and authentication.
2. A second SA applies the ESP protocol for encryption.
3. Commonly used when:
1. Both integrity and confidentiality are required without encapsulating the entire packet.
4. Advantages: Efficient, as it avoids the overhead of encapsulation.
2. Iterated Tunneling Multiple SAs are applied in tunnel mode, where one encapsulated packet becomes the
payload for the next.
• Example:
• An inner SA encrypts the packet with ESP in tunnel mode.
• An outer SA applies AH for authentication and integrity in tunnel mode.
• Commonly used in:
• Multi-hop VPNs or hierarchical networks requiring layered security.
Advantages: Each layer can be separately processed by different gateways or security devices.
3. Transport-Tunnel Combination A combination of transport mode and tunnel mode SAs applied to the same
packet.
• Example:
• An SA in transport mode provides end-to-end security for payload integrity and encryption.
• Another SA in tunnel mode secures the entire IP packet as it traverses an intermediate network.
• Commonly used in:
• Scenarios requiring end-to-end security between hosts while protecting data in transit across an insecure network.
• Advantages: Balances comprehensive security and operational flexibility.
How SA Combination is Implemented
• Security Policy Database (SPD): Maintains rules for determining the need for IPsec protection and
how multiple SAs are combined.
• SA Bundles: A set of SAs applied sequentially to a single IP packet. Each SA in the bundle may use
different protocols (AH, ESP) or modes (transport, tunnel).
Practical Applications of Combined SAs
1. VPNs with Layered Security:
1. Internal traffic between branch offices might use transport mode with ESP for confidentiality.
2. Outer layer uses tunnel mode with AH for secure communication between VPN gateways.
2. Hierarchical Networks:
1. Different administrative domains might apply their own security policies to the same packet using multiple
tunnel mode SAs.
3. Mobile Networks:
1. Transport mode SAs for secure host-to-host communications.
2. Tunnel mode SAs for secure routing between mobile devices and their home networks.
Key Management
• Key management refers to the
processes and procedures involved
in generating, storing, distributing,
and managing cryptographic keys
used in cryptographic algorithms
to protect sensitive data.
• It ensures that keys used to protect
sensitive data are kept safe from
unauthorized access or loss.
1. Key Generation:

• Creation: Keys are created using secure algorithms to ensure randomness and strength.

• Initialization: Keys are initialized with specific parameters required for their intended use (e.g., length, algorithm).

2. Key Distribution:

• Sharing: For symmetric keys, secure methods must be used to share the key between parties.

• Publication: For asymmetric keys, the public key is shared openly, while the private key remains confidential.

3. Key Storage:

• Protection: Keys must be stored securely, typically in hardware security modules (HSMs) or encrypted key stores, to prevent unauthorized
access.

• Access Control: Only authorized users or systems should be able to access keys.
4. Key Usage:

• Application: Keys are used for their intended cryptographic functions, such as encrypting/decrypting data or signing/verifying messages.

• Monitoring: Usage is monitored to detect any unusual or unauthorized activities.

5. Key Rotation:

• Updating: Keys are periodically updated to reduce the risk of exposure or compromise.

• Re-Keying: New keys are generated and distributed, replacing old ones while ensuring continuity of service.

6. Key Revocation:

• Invalidation: Keys that are no longer secure or needed are invalidated.

• Revocation Notices: For public keys, revocation certificates or notices are distributed to inform others that the key should no longer be
trusted.
7. Key Archival:
• Storage: Old keys are securely archived for future reference or compliance
purposes.

• Access Restrictions: Archived keys are kept in a secure location with restricted
access.

8. Key Destruction:
• Erasure: When keys are no longer needed, they are securely destroyed to prevent
any possibility of recovery.

• Verification: The destruction process is verified to ensure that no copies remain.

Introduction to Secure Socket Layer
SSL, or Secure Sockets Layer, is an Internet security protocol that encrypts data to keep it safe. It was
created by Netscape in 1995 to ensure privacy, authentication, and data integrity in online
communications. SSL is the older version of what we now call TLS (Transport Layer Security).

• Websites using SSL/TLS have “HTTPS” in their URL instead of “HTTP.”

How does SSL work?

• Encryption: SSL encrypts data transmitted over the web, ensuring privacy. If someone intercepts
the data, they will see only a jumble of characters that is nearly impossible to decode.

• Authentication: SSL starts an authentication process called a handshake between two devices to
confirm their identities, making sure both parties are who they claim to be.

• Data Integrity: SSL digitally signs data to ensure it hasn’t been tampered with, verifying that the
data received is exactly what was sent by the sender.

Additionally, SSL helps prevent cyber attacks by:

• Authenticating Web Servers: Ensuring that users are connecting to the legitimate website, not a
fake one set up by attackers.

• Preventing Data Tampering: Acting like a tamper-proof seal, SSL ensures that the data sent and
received hasn’t been altered during transit.
Secure Socket Layer Protocols

• SSL Record Protocol

• Handshake Protocol

• Change-Cipher Spec Protocol

• Alert Protocol
SSL Record Protocol
SSL Record provides two services to SSL
connection.
• Confidentiality

• Message Integrity

• In the SSL Record Protocol

application data is divided into
fragments. The fragment is
compressed and then encrypted MAC
(Message Authentication Code)
generated by algorithms like SHA
(Secure Hash Protocol) and MD5
(Message Digest) is appended.
• After that encryption of the data is
done and in last SSL header is
appended to the data.
Handshake Protocol
Handshake Protocol is used to establish sessions. This protocol
allows the client and server to authenticate each other by
sending a series of messages to each other. Handshake
protocol uses four phases to complete its cycle.

• Phase-1: In Phase-1 both Client and Server send hello-

packets to each other. In this IP session, cipher suite and
protocol version are exchanged for security purposes.

• Phase-2: Server sends his certificate and Server-key-

exchange. The server end phase-2 by sending the Server-
hello-end packet.

• Phase-3: In this phase, Client replies to the server by sending

his certificate and Client-exchange-key.

• Phase-4: In Phase-4 Change-cipher suite occurs and after

this the Handshake Protocol ends.
Change-Cipher Protocol & Alert Protocol
• Change Cipher Protocol This protocol
uses the SSL record protocol.
Unless Handshake Protocol is completed,
the SSL record Output will be in a pending
state. After the handshake protocol, the
Pending state is converted into the current
state.
Change-cipher protocol consists of a
single message which is 1 byte in length
and can have only one value. This
protocol’s purpose is to cause the pending
state to be copied into the current state.
Alert Protocol

• This protocol is used to convey SSL-related

alerts to the peer entity. Each message in
this protocol contains 2 bytes.
Secure electronic transaction (SET)
• Secure Electronic Transaction or SET is a
security protocol designed to ensure the
security and integrity of electronic
transactions conducted using credit cards.
Unlike a payment system, SET operates as a
security protocol applied to those payments.
It uses different encryption and hashing
techniques to secure payments over the
internet done through credit cards. The SET
protocol was supported in development by
major organizations like Visa, Mastercard,
and Microsoft which provided its Secure
Transaction Technology (STT), and Netscape
which provided the technology of Secure
Socket Layer (SSL).
SET functionalities:
• Provide Authentication
• Merchant Authentication – To prevent theft, SET allows customers to check previous
relationships between merchants and financial institutions. Standard X.509V3 certificates
are used for this verification.
• Customer / Cardholder Authentication – SET checks if the use of a credit card is done
by an authorized user or not using X.509V3 certificates.
• Provide Message Confidentiality: Confidentiality refers to preventing
unintended people from reading the message being transferred. SET implements
confidentiality by using encryption techniques. Traditionally DES is used for
encryption purposes.

• Provide Message Integrity: SET doesn’t allow message modification with the
help of signatures. Messages are protected against unauthorized modification
using RSA digital signatures with SHA-1 and some using HMAC with SHA-1,
Requirements in SET: The SET protocol has some requirements to meet, some of the important requirements are:
• It has to provide mutual authentication i.e., customer (or cardholder) authentication by confirming if the customer is an intended user or
not, and merchant authentication.

• It has to keep the PI (Payment Information) and OI (Order Information) confidential by appropriate encryptions.

• It has to be resistive against message modifications i.e., no changes should be allowed in the content being transmitted.

• SET also needs to provide interoperability and make use of the best security mechanisms.

Participants in SET: In the general scenario of online transactions, SET includes similar participants:
1. Cardholder – customer

2. Issuer – customer financial institution

3. Merchant

4. Acquirer – Merchant financial

5. Certificate authority – Authority that follows certain standards and issues certificates(like X.509V3) to all other participants.
System Security: Introductory idea of Intrusion
System security refers to the protection of information systems from theft, damage, disruption, or unauthorized access.
• It encompasses tools, techniques, and practices to safeguard hardware, software, data, and networks. Below are introductory
ideas of key concepts related to system security.
Intrusion
• Intrusion refers to unauthorized access or manipulation of a system or network, typically with malicious intent. Intrusions can
result in data theft, service disruption, or system damage.
Types of Intrusions
• Passive Intrusion:
• Involves monitoring system activities without affecting its operation.
• Example: Eavesdropping on network traffic (sniffing).
• Active Intrusion:
• Involves modifying or disrupting the system.
• Example: Exploiting software vulnerabilities to gain unauthorized access.
Common Methods of Intrusion
• Exploiting weak passwords or credentials.
• Utilizing software bugs or misconfigurations.
• Social engineering techniques, such as phishing.
Intrusion Detection
Intrusion Detection Systems (IDS) are tools or mechanisms designed to identify and respond to potential intrusions. They help detect abnormal activities
and alert administrators to take action.
• Types of IDS
1. Network-Based IDS (NIDS):
1. Monitors network traffic for suspicious activities.
2. Example: Detecting unusual spikes in traffic or unauthorized protocol usage.
2. Host-Based IDS (HIDS):
1. Monitors activities on a specific host or device.
2. Example: Tracking changes in system files or unusual application behavior.
Detection Techniques
• Signature-Based Detection:
• Matches patterns of known attack signatures.
• Effective for well-documented threats but cannot detect novel attacks.
• Anomaly-Based Detection:
• Detects deviations from normal behavior.
• Useful for identifying new and emerging threats but may generate false positives.
Response Mechanisms
• Alerting: Notifies administrators of detected intrusions.
• Automated Actions: Blocks malicious IPs, terminates suspicious processes, or isolates affected systems.
Viruses and Related Threats
Viruses
• A virus is a malicious program that attaches itself to a host file or program and replicates when executed.
• Characteristics:
• Requires human interaction to spread (e.g., opening an infected file).
• Can cause data corruption, system slowdowns, or total failure.
Related Threats
1. Worms:
1. Self-replicating programs that spread across networks without user intervention.
2. Example: The 2001 Code Red worm.
2. Trojan Horses:
1. Malicious software disguised as legitimate applications.
2. Often used to create backdoors for attackers.
3. Spyware:
1. Collects sensitive information without user consent.
2. Example: Monitoring browsing habits or capturing keystrokes (keyloggers).
4. Ransomware:
1. Encrypts user data and demands payment for decryption keys.
2. Example: WannaCry attack in 2017.
5. Rootkits:
1. Conceals malicious activities or unauthorized access by modifying system files.
Firewalls
A firewall is a network security device or software designed to monitor and control incoming and
outgoing traffic based on predetermined security rules.
• Types of Firewalls
1. Packet-Filtering Firewalls:
1. Examines packet headers (source and destination IP addresses, ports, protocols).
2. Allows or blocks packets based on access control rules.
2. Stateful Inspection Firewalls:
1. Tracks the state of active connections and makes decisions based on the context.
2. Offers better security than packet-filtering firewalls.
3. Proxy Firewalls:
1. Acts as an intermediary between internal users and external services.
2. Filters requests and responses for enhanced security.
4. Next-Generation Firewalls (NGFW):
1. Incorporates features like deep packet inspection, intrusion prevention, and application-layer filtering.
Key Features
• Access Control: Defines which traffic is permitted or denied.
• Network Address Translation (NAT): Hides internal network structure.
• Logging and Monitoring: Tracks traffic for analysis and audits.
Functions
• Prevents unauthorized access.
• Protects against known threats like port scanning or brute force attacks.
• Controls access to malicious websites or services.
Importance of System Security
• Prevents data breaches: Protects sensitive data from being accessed or stolen.
• Ensures system availability: Keeps systems operational and reduces downtime.
• Builds user trust: Demonstrates a commitment to safeguarding information.
• Complies with regulations: Helps meet legal and industry-specific security standards.