IPSec (Internet Protocol Security) — Securing IP Communication¶

Internet Protocol Security (IPSec) is a suite of network security protocols that provides authentication, integrity, confidentiality, and secure communication for IP packets. IPSec is widely used to build Site-to-Site Virtual Private Networks (VPNs), Remote Access VPNs, and Hybrid Cloud Connectivity. By encrypting network traffic at the IP layer, IPSec protects data from interception, modification, and spoofing while it travels across untrusted networks such as the Internet. Every Linux administrator, DevOps engineer, Cloud Architect, Platform Engineer, Site Reliability Engineer (SRE), and Network Engineer should understand IPSec.

Learning Path¶

Networking Mastery → Module 8: Network Security → Lesson 2

Difficulty: Intermediate

Reading Time: 120 Minutes

Course Progress

Course: Networking Mastery

Module: Network Security

Lesson: 2 of 9

What You'll Learn¶

After completing this lesson, you'll be able to:

Understand IPSec
Learn IPSec architecture
Understand Authentication Header (AH)
Learn Encapsulating Security Payload (ESP)
Compare Tunnel Mode and Transport Mode
Understand Internet Key Exchange (IKE)
Apply IPSec in enterprise and cloud environments

Prerequisites¶

Complete:

VPN

Why Learn IPSec?¶

Imagine two offices communicating over the Internet.

Without IPSec:

Branch Office

↓

Internet

↓

Head Office

Traffic could potentially be intercepted.

With IPSec:

Branch Office

↓

Encrypted Tunnel

↓

Internet

↓

Encrypted Tunnel

↓

Head Office

Communication is encrypted and authenticated.

What is IPSec?¶

IPSec (Internet Protocol Security) is a suite of protocols that secures IP communication.

It provides:

Authentication
Encryption
Data Integrity
Replay Protection

at the Network Layer (Layer 3) of the Open Systems Interconnection (OSI) model.

Why Use IPSec?¶

Organisations use IPSec to:

Build Site-to-Site VPNs
Secure Remote Access
Connect Data Centres
Connect Branch Offices
Enable Hybrid Cloud Connectivity
Protect Sensitive Data

IPSec Architecture¶

Application

↓

IP Layer

↓

IPSec

↓

Internet

↓

IPSec

↓

Destination

IPSec protects packets before they leave the system.

IPSec Services¶

IPSec provides four major security services:

Confidentiality
Integrity
Authentication
Anti-Replay Protection

Confidentiality¶

Confidentiality ensures:

Unauthorized Users

Cannot

Read Data

Encryption protects packet contents.

Integrity¶

Integrity verifies:

Data

Not Modified

During Transit

Recipients can detect unauthorised changes.

Authentication¶

Authentication verifies:

Sender

Identity

It helps prevent impersonation attacks.

Anti-Replay Protection¶

Attackers may attempt to resend captured packets.

IPSec uses:

Sequence Numbers

to detect and reject replayed packets.

IPSec Components¶

The primary IPSec protocols are:

Authentication Header (AH)
Encapsulating Security Payload (ESP)

Authentication Header (AH)¶

AH provides:

Authentication
Integrity
Anti-Replay Protection

AH does not encrypt packet payloads.

Example:

Packet

↓

AH

↓

Internet

Packet contents remain visible.

Encapsulating Security Payload (ESP)¶

ESP provides:

Encryption
Authentication (optional in the protocol design, commonly used in practice)
Integrity
Anti-Replay Protection

Example:

Packet

↓

ESP Encryption

↓

Internet

Packet contents are protected from unauthorised viewing.

AH vs ESP¶

AH	ESP
Authentication	Encryption
Integrity	Integrity
Anti-Replay	Anti-Replay
No Confidentiality	Provides Confidentiality

In modern deployments, ESP is used far more frequently than AH.

IPSec Modes¶

IPSec supports:

Transport Mode
Tunnel Mode

Transport Mode¶

Protects:

Payload Only

Original IP header remains visible.

Example:

IP Header

↓

ESP

↓

Encrypted Payload

Typically used for host-to-host communication.

Tunnel Mode¶

Protects:

Entire Original Packet

A new IP header is added.

Example:

New IP Header

↓

Encrypted Original Packet

Tunnel Mode is the most common choice for Site-to-Site VPNs.

Transport vs Tunnel Mode¶

Transport Mode	Tunnel Mode
Encrypts Payload	Encrypts Entire Original Packet
Host-to-Host	Gateway-to-Gateway
Lower Overhead	Better Network Protection
Original Header Visible	Original Header Hidden

Internet Key Exchange (IKE)¶

Before encryption begins:

Two Devices

↓

Authenticate

↓

Exchange Keys

↓

Create Secure Tunnel

This process is handled by:

IKE

IKE Phases¶

Modern IPSec commonly uses:

IKE Phase 1 (Establish Secure Management Channel)
IKE Phase 2 (Negotiate IPSec Security Associations)

With IKEv2, the process is streamlined while providing the same overall goals.

Security Association (SA)¶

A:

Security Association

defines:

Encryption Algorithm
Authentication Algorithm
Keys
Lifetime

Each IPSec connection establishes one or more Security Associations.

IPSec Workflow¶

Authenticate

↓

Exchange Keys

↓

Create Tunnel

↓

Encrypt Traffic

↓

Transmit

↓

Decrypt

↓

Deliver

Enterprise Example¶

Branch Office

↓

Firewall

↓

IPSec Tunnel

↓

Internet

↓

Firewall

↓

Head Office

Both offices communicate securely over the public Internet.

Hybrid Cloud Example¶

On-Premises

↓

IPSec VPN

↓

Cloud VPN Gateway

↓

Cloud Network

Organisations securely connect data centres to cloud environments.

Cloud Perspective¶

Major cloud providers support IPSec VPNs.

Common use cases:

Hybrid Cloud
Branch Connectivity
Disaster Recovery
Multi-Cloud Networking

Cloud VPN gateways commonly use IPSec Tunnel Mode.

Kubernetes Perspective¶

IPSec may be used to:

Secure communication between Kubernetes clusters
Protect hybrid Kubernetes deployments
Encrypt traffic between on-premises and cloud clusters

Some Container Network Interface (CNI) plugins also support IPSec encryption between nodes.

Linux Perspective¶

Linux supports IPSec through implementations such as:

strongSwan
Libreswan

Display network interfaces.

ip addr

Display routing table.

ip route

Display IPSec Security Associations.

ip xfrm state

Display IPSec policies.

ip xfrm policy

IPSec Packet Flow¶

Application

↓

IPSec

↓

Encrypt

↓

Internet

↓

Decrypt

↓

Application

Advantages of IPSec¶

Strong Encryption
Data Integrity
Authentication
Anti-Replay Protection
Transparent to Applications
Widely Supported

Limitations¶

More complex to configure than some application-layer VPNs
Encryption introduces processing overhead
Network Address Translation (NAT) traversal requires additional mechanisms in many deployments
Key management must be carefully maintained

Hands-on Lab¶

Task 1¶

Display IP configuration.

ip addr

Task 2¶

Display routing table.

ip route

Task 3¶

Display IPSec Security Associations.

ip xfrm state

Task 4¶

Display IPSec policies.

ip xfrm policy

Task 5¶

Compare:

AH
ESP

Task 6¶

Compare:

Transport Mode
Tunnel Mode

Task 7¶

Draw a Site-to-Site IPSec VPN architecture connecting two offices.

Task 8¶

Research IPSec implementations on:

Linux
Cisco
Amazon Web Services (AWS)
Microsoft Azure
Google Cloud

Compare their deployment models.

Linux Commands¶

Command	Purpose
`ip addr`	Display IP configuration
`ip route`	Display routing table
`ip xfrm state`	Display IPSec Security Associations
`ip xfrm policy`	Display IPSec policies
`ss -tun`	Display active network connections

Common Mistakes¶

❌ Confusing AH with ESP.

✅ Remember that ESP provides encryption while AH does not.

❌ Using Transport Mode for gateway-to-gateway VPNs.

✅ Use Tunnel Mode for Site-to-Site VPNs.

❌ Forgetting key negotiation.

✅ Verify IKE configuration and authentication.

❌ Ignoring Security Association lifetimes.

✅ Monitor and renew SAs as required.

❌ Assuming IPSec alone secures applications.

✅ Combine IPSec with firewalls, identity controls, and monitoring.

Best Practices¶

Prefer ESP for modern IPSec deployments.
Use IKEv2 where supported.
Protect VPN gateways with firewalls.
Rotate cryptographic keys regularly.
Monitor tunnel health and Security Associations.
Document VPN topology and security policies.
Use strong authentication methods such as certificates.

Interview Questions¶

Beginner¶

What is IPSec?
What security services does IPSec provide?
What is the difference between AH and ESP?
What is a Security Association?

Intermediate¶

Compare Transport Mode and Tunnel Mode.
Explain the role of IKE.
Why is ESP more commonly used than AH?
How does IPSec protect against replay attacks?

Architect Level¶

Design an IPSec architecture connecting multiple branch offices.
Explain IPSec deployment for hybrid cloud environments.
How would you troubleshoot an IPSec tunnel that fails to establish?

Summary¶

In this lesson, you learned:

IPSec
Authentication Header (AH)
Encapsulating Security Payload (ESP)
Transport Mode
Tunnel Mode
Internet Key Exchange (IKE)
Security Associations (SA)
Enterprise IPSec VPNs
Linux IPSec Commands

IPSec is the industry-standard protocol suite for securing IP communication. By providing encryption, authentication, integrity, and replay protection at the network layer, IPSec enables secure VPN connectivity between users, branch offices, data centres, and cloud environments. Its widespread adoption makes it a foundational technology for enterprise networking and hybrid cloud security.

Key Takeaways¶

IPSec secures IP communication at Layer 3.
It provides confidentiality, integrity, authentication, and replay protection.
ESP is the most commonly used IPSec protocol because it supports encryption.
Tunnel Mode is typically used for Site-to-Site VPNs.
IKE establishes secure tunnels and negotiates cryptographic parameters.
IPSec is widely used across enterprise, cloud, and hybrid networking environments.

What's Next?¶

SSL/TLS

In the next lesson, you'll learn about SSL/TLS (Secure Sockets Layer / Transport Layer Security).

You'll explore:

What SSL/TLS is
SSL vs TLS
TLS Handshake
Digital Certificates
Public Key Infrastructure (PKI)
HTTPS
TLS Best Practices

By the end of the lesson, you'll understand how TLS secures web applications, APIs, email, and modern Internet communications using encryption, certificates, and authenticated key exchange.