PCIe

Learn simply from Basics

A Collection of Key
Concepts & Best Practices

Prasanthi Chanda
 TABLE OF CONTENTS

CHAPTER 1 : Introduction to PCIe

CHAPTER 2 : PCIe Architecture and

Protocol Stack

CHAPTER 3: PCIe Link Training and

Initialization

CHAPTER 4: PCIe Transaction Layer

Protocol

CHAPTER 5: PCIe Configuration Space

CHAPTER 6: PCIe Interrupts and Power

Management

CHAPTER 7: PCIe Error Handling and

Debugging

CHAPTER 8: PCIe Virtualization and

SR-IOV

CHAPTER 9: PCIe in Real-World

Applications
 CHAPTER 1
Introduction to PCIe

What is PCIe? Evolution from PCI to PCIe

PCI Express (Peripheral Component Interconnect Express), officially
abbreviated as PCIe or PCI-E is a high-speed serial computer expansion
bus standard, meant to replace the older PCI, PCI-X and AGP bus
standards.
It is the common motherboard interface for personal computers'
graphics cards, capture cards, sound cards, hard disk drive host
adapters, SSDs, Wi-Fi, and Ethernet hardware connections.

Fig.1 PCIe Logo

The evolution of PCIe (Peripheral Component Interconnect Express)

standards over time, showing the increasing data transfer rates in
GT/s (Gigatransfers per second) from PCIe 1.0 to PCIe 6.0. Here’s a
breakdown of the progression:
PCIe 1.0 (2003) – Introduced with a data transfer rate of 2.5 GT/s.
PCIe 2.0 (2006) – Doubled the transfer rate to 5 GT/s.
PCIe 3.0 (2010) – Increased to 8 GT/s, continuing the trend of doubling
bandwidth every generation.
PCIe 4.0 (2017)– Further improved to 16 GT/s, reflecting advancements
in high-speed computing.
PCIe 5.0 (2019) – Reached 32 GT/s, supporting faster data movement
crucial for AI, ML, and high-performance applications.
PCIe 6.0 (2022) – Achieved 64 GT/s, marking a significant leap in
performance.
The graph demonstrates an exponential increase in data transfer
speeds, enabling higher bandwidth for modern computing needs such
as AI acceleration, cloud computing, and high-speed storage
solutions.

Fig.2, 3 PCIe specification evolution through six generations

spanning three decades
 Key advantages over previous interfaces

PCIe offers several key advantages over previous interfaces like PCI
and AGP, as well as over earlier PCIe versions. These include:
Higher Bandwidth – Each generation of PCIe has doubled the
data transfer rate, with PCIe 6.0 reaching 64 GT/s, significantly
improving data throughput for high-performance applications.
Lower Latency – PCIe reduces communication delays,
improving real-time processing performance, which is critical
for AI, ML, and networking applications.
Scalability – PCIe supports multiple lane configurations (x1, x4,
x8, x16), allowing flexible bandwidth allocation based on
application needs.
Power Efficiency – Newer PCIe versions optimize power
consumption using techniques like PAM-4 signaling (PCIe 6.0)
while maintaining high speeds.
Backward Compatibility – PCIe maintains compatibility with
older versions, allowing new devices to work with previous-
generation slots.
Improved Reliability – Features like Forward Error Correction
(FEC) in PCIe 6.0 enhance data integrity and signal reliability.
Better Support for Emerging Technologies – Higher
bandwidth and lower latency enable PCIe to handle
advancements in GPUs, SSDs, AI accelerators, and networking
(e.g., 400G Ethernet).

These advantages make PCIe the preferred interface for modern

high-speed computing applications.
 PCIe architecture overview

PCIe (Peripheral Component Interconnect Express) is a high-speed

serial communication protocol used to connect peripheral devices
to the CPU and memory. The image represents a hierarchical PCIe
architecture, which consists of the following key components:
1. Root Complex (RC)
The Root Complex (RC) is the central controller that connects
the CPU and memory to the PCIe devices.
It manages data flow between system memory and PCIe
endpoints.
The RC contains multiple PCIe ports to communicate with
different devices.

Fig. 4 Example of the PCI Express topology: white "junction

boxes" represent PCI Express device downstream ports, while
the gray ones represent upstream ports.
2. System Memory
The memory is connected to the CPU and serves as the
primary storage for application execution.
PCIe devices, such as GPUs and SSDs, access memory through
the Root Complex to read/write data efficiently.
3. PCIe Endpoints (Devices)
PCIe endpointsare peripheral devices that use PCIe for
communication, such as:
Graphics Processing Units (GPUs)
Solid-State Drives (SSDs)
Network Interface Cards (NICs)
Sound Cards, AI Accelerators, etc.
These devices are directly connected to the Root Complex or
through a PCIe switch.
4. PCIe Switch
A PCIe switch extends connectivity when multiple devices
need to share a single PCIe connection.
It functions like a network switch, allowing multiple PCIe
endpoints to communicate with the CPU efficiently.
The switch can provide lane bifurcation, dynamically
assigning bandwidth to different devices.
5. PCIe Bridge (PCIe to PCI/PCI-X)
A PCIe bridge connects older PCI or PCI-X devices to a modern
PCIe system.
It ensures backward compatibility, allowing legacy devices to
function with newer PCIe-based architectures.
6. Legacy Endpoint
A legacy endpoint is an older PCIe-compatible device that does
not support some advanced PCIe features (e.g., power
management or advanced error reporting).
These devices still work within the PCIe hierarchy but may
require compatibility modes.
 CHAPTER 2
PCIe Architecture and
Protocol Stack

PCIe (Peripheral Component Interconnect Express) follows a

layered architecture to ensure efficient communication between
devices. It consists of the following four key layers:
1. Transaction Layer (TL)
The topmost layer that handles data packetization and request
processing.
Communicates with the operating system and application
software.
Key responsibilities:
Packet Formation: Converts data into Transaction Layer
Packets (TLPs).
Flow Control: Ensures efficient data transmission without
congestion.
Transaction Types:
Memory Read/Write
I/O Read/Write
Configuration Read/Write
Message Transactions
2. Data Link Layer (DLL)
Ensures error-free data transfer between PCIe devices.
Adds Sequence Numbers & CRC (Cyclic Redundancy Check) for
data integrity.
 Key responsibilities:
Acknowledge/NACK Mechanism: If an error is detected, data
is resent.
Creates Data Link Layer Packets (DLLPs).
Manages Flow Control & Retransmission.
3. Physical Layer (PL)
Handles electrical signaling and actual data transmission over
PCIe lanes.
Consists of transceivers that convert digital signals into physical
electrical signals.
Key responsibilities:
Encoding/Decoding: Uses 8b/10b (PCIe 1.0-2.0) or 128b/130b
(PCIe 3.0+) encoding to maintain signal integrity.
Lane Management: Supports x1, x4, x8, x16 lane configurations
for scalable bandwidth.
Synchronization: Maintains link stability using a clocking
mechanism.

Fig. 5 PCIe Protocol Layers

 CHAPTER 3
PCIe Link Training and
Initialization
PCIe (Peripheral Component Interconnect Express) linking and
initialization is the process of establishing a reliable
communication link between PCIe devices (e.g., CPU, GPU, SSD,
or NIC).
It ensures proper data transmission, error handling, and power
management. The initialization sequence follows a well-defined
Link Training and Status State Machine (LTSSM).

Fig 6. Link Training and Status State Machine, LTSSM

Detect:
The devices start the Rx detect circuit on each lane that allows
the device to determine if the device has a link partner to pair
with.
Assuming that the PCIe Rx detect circuit sees the other device,
each individual lane begins to transmit serial data at 2.5 Gbps
(Gen1).
Detect substate machine: 12ms timeout between detect substate.

Polling:
Root complex, retimer and the endpoint all begin transmitting
ordered sets of data called training sequences at PCIe Gen 1 speeds
in order to establish bit lock and symbol lock.
Bit lock: when the receiver locks the clock frequency of the
transmitter.
Symbol lock: when the receiver is able to decode the valid symbol
coming from the transmitter.
Polling substate machine:
24ms timeout during Polling.Active state, it will back to detcet
state.
48ms timeout during Polling.configuration state, it will back
to detcet state.
Configuration:
Determine link width and lane numbers
Configuration substate machine:
24ms timeout during Configuration.LinkWidth.Start
state, it will back to detect state.
2ms timeout during other states, it will back to detect
state.
 CHAPTER 4
PCIe Transaction Layer
Protocol
The Transaction Layer uses TLPs to communicate request and
completion data with other PCI Express devices. TLPs may address
several address spaces and have a variety of purposes. Each TLP
has a header associated with it to identify the type of transaction.

Transaction Layer Packet

A TLP consists of a header, an optional data payload, and an

optional TLP digest. The Transaction Layer generates outgoing
TLPs based on the information it receives from its device core. The
Transaction Layer then passes the TLP on to its Data Link Layer
for further processing. The Transaction Layer also accepts
incoming TLPs from its Data Link Layer.
TLP Headers
All TLPs consist of a header that contains the basic identifying
information for the transaction. The TLP header may be either 3
or 4 DWords in length, depending on the type of transaction.
The format of first DWord is shown in the figure below

TLP Handling

A TLP that makes it through the Data Link Layer has been
verified to have traversed the link properly, but that does not
necessarily mean that the TLP is correct. A TLP may make it
across the link intact, but may have been improperly formed by
its originator. As such, the receiver side of the Transaction
Layer performs some checks on the TLP to make sure it has
followed the rules. If the incoming TLP does not check out
properly, it is considered a malformed packet, is discarded
(without updating receiver flow control information) and
generates an error condition. If the TLP is legitimate, the
Transaction Layer updates its flow control tracking and
continues to process the packet
Request Handling
If the TLP is a request packet, the Transaction Layer first checks to
make sure that the request type is supported. If it is not supported, it
generates a non-fatal error and notifies the root complex.

Completion Handling
If a device receives a completion that does not correspond to any
outstanding request, that completion is referred to as an unexpected
completion. Receipt of an unexpected completion causes the
completion to be discarded and results in an error-condition
(nonfatal).
The receipt of unsuccessful completion packets generates an error
condition that is dependent on the completion status.
 CHAPTER 5
PCIe Configuration Space
The configuration space for each link is where driver software can
inspect the capabilities and status advertised by the device and to set
certain parameters. The configuration space access TLPs, are of two
types of spaces—type 0 and type 1, with corresponding configuration
TLP types. The former is for endpoints, and the latter for root
complexes and switches that have virtual PCI-PCI bridges.

Both type 0 and type 1 configurations have a set of common registers

in the PCI compatible region (0 to 3Fh). The diagram below shows
these common registers and their relative position in the
configuration space.
Type 0 Configuration Space
Type 0 configuration spaces are for endpoints. Beyond the
common registers described above, this type of configurations
space is mainly given over to defining base address registers
(BARs), but with a few extra registers thrown in. The diagram
below shows the layout of the PCI compatible region for a type 0
configuration space.
There are six base address registers which are used to define
regions of memory mapping that the device can be assigned. This
can be up to six individual 32 bits address regions, or even/odd
pairs can be formed for 64-bit address regions. The lower 4 bits
define characteristics of the address:
Bit 0 is region type: 0 = memory, 1 = I/O
Bits 2:1 is locatable type (memory only): 0 = any 32-bit region, 1 = <
1MB, 2 = any 64-bit region
Bit 3 is prefetchable flag (memory only): 0 = not prefetchable, 1 =
prefetchable
Type 1 Configuration Space
The type 1 configuration space is for switches and root complexes
—basically devices with virtual PCI-PCI bridges. The diagram
below shows the layout of the PCI compatible region for a type 1
configuration space.
Type 1 spaces also have base address registers for mapping into the
address space, but it’s limited to two 32-bit regions or a single 64-bit
region. The primary base number is not used in PCIe but must
exists as a read/write register for legacy software. The secondary
bus number is the bus number immediately downstream of the
virtual PCI-PCI bridge, whilst the subordinate bus number is the
highest bus number of all the busses that are reachable
downstream. These, then, are used construct the bus hierarchy and
to route packets that use ID routing.

The type 1 configurations space registers also have a set of

base/limit pairs, split over multiple registers, which define an
upper and lower boundary for the memory and I/O regions. The
memory region is split into non-prefetchable and prefetchable
regions (see above). If a TLP is received by the link from upstream,
and it fits between the base and limit values (for the relevant type)
of the link, it will be forwarded on that downstream port.
 CHAPTER 6
PCIe Interrupts and Power
Management

The power management capability has an ID of 01h. A diagram

of this structure is shown below:

The power management capability structure, after the common ID

and next capability pointer, has a power management capabilities
register (PMC) which is a set of read-only fields indicating whether a
PME clock is required for generating power management events
(hardwired to 0 for PCIe), whether device specific initialisation is
required, the maximum 3.3V auxiliary current required, whether D1
and D2 power states are supported and which power states can assert
the power management event signal. In addition, this structure has a
power management control/status register (PMCSR).
A two-bit field indicates what the current power state is (D3hot to
D0) or, when written to, set to a new power state. Another bit
indicates whether a reset is performed when software changes
the power state to D0. Power management events can be enabled.
A data select field selects what information is shown in the data
register (the last byte of the power management capabilities
structure) if that register is implemented. This optional
information is for reporting power dissipated or consumed for
each of the power states. Finally there is a PME_Status bit that
shows the state of the PME# signal, regardless of whether PME is
enabled or not.

After the power management control/status register is a bridge

support extension register (PMCSR_BSE) that only has two bits. A
read-only B2_B3# bit determines the action when transitioning to
D3hot. When set, the secondary busses clock will be stopped.
When clear, the secondary bus has bower removed. A BPCC_En
bit indicates that the bus power/clock control features are
enabled or not. When not enabled the power control bits in the
power management control/status register can’t be used by
software to control the power or clock of the secondary bus.
 CHAPTER 7
PCIe Error Handling and
Debugging
Advanced error reporting gives more granularity and control over
specific errors than the default functionality. For example,
differentiation between uncorrectable errors can be made for
masking, status, and control of severity (i.e. fatal/non-fatal). For
example differentiation can be made between malformed TLP and
ECRC errors. Status and mask registers exist for correctable errors
(though no severity register). A control and capabilities register
gives status and control over ECRC. A set of header log registers
capture the TPL header of a reported error. For root ports and root
complex event collectors, a set of registers enable/disable
correctable, non-fatal and fatal errors, and report error reception, as
well as logging the source (requester ID) of an error and its level. The
diagram below gives an overview of the advance error reporting
capabilities structure.
 CHAPTER 8
PCIe Virtualization and
SR-IOV
PCIe Virtualization, specifically through the "Single Root I/O
Virtualization" (SR-IOV) standard, allows a single physical PCIe
device (like a network adapter) to present itself as multiple
virtual devices, enabling direct access to its resources by
multiple virtual machines (VMs) on a server, significantly
improving network performance for each VM by bypassing the
hypervisor overhead; essentially, a single physical device can
act as multiple virtual devices with separate resources, each
accessible by a different VM.
Key points about SR-IOV:
Physical Function (PF):
The main PCIe function of the device, which manages and
configures the virtual functions.
Virtual Functions (VFs):
Lightweight PCIe functions created by the PF, each acting
as a separate virtual device accessible by a VM.
Benefits:
Improved performance: VMs can directly access the network
card, reducing latency and increasing throughput.
Scalability: Allows a single physical device to be shared among
multiple VMs.
Efficient resource utilization: Reduces the need for dedicated
network cards for each VM.
 CHAPTER 9
PCIe in Real-World
Applications
PCIe (Peripheral Component Interconnect Express) is widely
used in various real-world applications, including:
1. High-Performance Computing (HPC) & AI
Used in GPUs (NVIDIA, AMD) for machine learning and AI
acceleration.
High-speed NVMe SSDs for faster data access in
supercomputers.
FPGA-based accelerators for data centers and AI inference.
2. Storage & Data Centers
NVMe SSDs connected via PCIe for ultra-fast data transfer.
PCIe switches for multi-device storage expansion.
RAID controllers for high-speed data redundancy and backup.

3. Networking & Communication

High-speed Ethernet (10G, 25G, 100G NICs) use PCIe for low-
latency networking.
5G and telecom infrastructure use PCIe for data processing.
InfiniBand and RDMA over PCIe for cloud computing and
high-frequency trading.

4. Consumer Electronics & Gaming

Graphics Cards (GPUs) in gaming PCs and workstations.
PCIe Sound Cards for professional audio processing.
Capture Cards for live streaming and video production.

5. Embedded Systems & Automotive

PCIe used in ADAS (Advanced Driver Assistance Systems) for
real-time processing.
Automotive ECUs and AI-based infotainment systems.
Industrial automation and IoT devices with PCIe-based AI
chips.

6. Medical & Scientific Applications

PCIe-based imaging cards in medical diagnostics (MRI, CT
scan).
High-speed data acquisition systems for scientific research.
PCIe connectivity in robotic surgery and biomedical devices.
 Excellence in World class
VLSI Training & Placements

Do follow for updates & enquires

+91- 9182280927