Distributed Architecture &

Intensive Computing
Presented by Dr. A. Djenadi

1
 Course objectives
After completing this course, you should be able to
 Identify various mechanisms for optimizing the performance of an architecture.
 Define the concept of parallelism, task distribution, and the sharing of data and memory among
multiple processors or cores.
 Measure the performance of different machines based on the parameters of their architectural
components.
 Compare different architectures and be able to choose an architecture for a given application.
 Make trade-offs between different parameters of an architecture (CPI, memory size, pipeline stages).
 Parallelize and execute an application on a parallel architecture using parallelization tools and
languages.

2
CHAPTER 1: FOUNDATIONS OF CONVENTIONAL
ARCHITECTURES

3
 Evolution of the computer
Computer technology has undergone remarkable advancements
in the approximately 90 years since the introduction of the first
general-purpose electronic computer.

Today, less than DA 20000 will purchase a smartphone that has

as much performance as the world’s fastest computer bought in
1993 for $50 million.

This rapid improvement is due to advances in the technology

used to build computers and innovations in computer design.

4
 Evolution of the computer
Timeline of the computer architecture evolution

• 1940s: Vacuum Tubes:

o The first electronic computers, such as the ENIAC (Electronic Numerical Integrator
and Computer) (1946), used vacuum tubes for calculations.
.

ENIAC (1946): USA army/ Pennsylvania

university uses decimal architecture, 5000
additions/sec, 330 multiplications/sec, 17,468
vacuum tubes, Weight :30 tones; energie 150 KW

5
 Evolution of the computer
Timeline of the computer architecture evolution
 1950s: Transistors era
o Transistors were later introduced in the 1950s, enabling miniaturization and
increased reliability.
o Assembly language and high-level programming languages were introduced.

6
 Evolution of the computer
Timeline of the computer architecture evolution
 1960s: Integrated circuits (ICs) and Von
Neumann Architecture
o Integrated circuits (ICs) allowed for even
smaller and more powerful computers.
o The Von Neumann architecture,
proposed by John von Neumann. It
separates memory and processing units,
allowing the loading of programs stored in
memory.

7
 Evolution of the computer
Timeline of the computer architecture evolution
• 1970s: Early Microprocessors:
o The introduction of the first microprocessors, such as the Intel 4004 (1971) and the Intel 8008
(1972), marks the beginning of the era of single-chip processors.

Intel 4004 (1971):

• Architecture: 4-bit
• Clock Speed: 740 kHz
• Instruction Set: 46 instructions
• Memory Capacity: 2,048 bytes
• Applications: Initially designed for calculators, used in various
devices
• Follow-up Microprocessors: Intel 4040 series, set the stage for x86
architecture

8
 Evolution of the computer
Timeline of the computer architecture evolution
• 1980s & 1990s: RISC vs CISC
o The 1980s saw the emergence of Reduced Instruction Set Computing (RISC) architectures.
This was in contrast to Complex Instruction Set Computing (CISC) architectures like x86.
o Parallel processing and multi-core architectures began to be explored.
o Moore's Law: Transistor count on chips doubles roughly every 2 years, driving performance
improvements.
o Pentium processors and PowerPC architecture were notable.

9
 Evolution of the computer
Timeline of the computer architecture evolution
Intel Pentium III (1999)
• Architecture: 32-bit
• Clock Speed: Ranged from 450 MHz to 1.4 GHz
• Cache: 16 KB Level 1 cache, up to 512 KB Level 2 cache
• Instruction Set: Intel A-32 (x86)
• Manufacturing Process: 0.25-micron to 0.13-micron
technology
• Transistors: Approximately 24 million
• Bus Speed: 100 MHz to 133 MHz

10
 Evolution of the computer
Timeline of the computer architecture evolution

Moore's Law: an observation and prediction made by Gordon Moore, co-founder of Intel Corporation, in
1965. The law states that the number of transistors on a microchip, and by extension, computing power,
tends to double approximately every two years while the cost per transistor decreases. In simpler terms,
this suggests that the performance and capacity of computer chips will increase exponentially over time.

11
 Evolution of the computer
Timeline of the computer architecture evolution

12
 Evolution of the computer
Timeline of the computer architecture evolution
Moore's Law limitation :

• Physical Limits (transistor miniaturization) : As transistors shrink, approaching the size of

individual atoms,
• Manufacturing complexity : Producing chips with billions of transistors at nanoscale is incredibly
complex and expensive
• Cost : The research and development required to push the boundaries of Moore's Law is becoming
increasingly costly
• Focus on specialization: the focus might shift towards specialized architectures and software
optimization for specific tasks, like artificial intelligence or machine learning.

13
 Evolution of the computer
Timeline of the computer architecture evolution
• 2000s & 2010s: Multicore Processors, Multiprocessors and Supercomputers
o Multiprocessors and multicore processors become common.
o Cluster computing
o Internet of Things (IoT)
o Parallel architectures / Parallel processing

14
 Evolution of the computer
Timeline of the computer architecture evolution

15
 Evolution of the computer
Timeline of the computer architecture evolution
Intel Core i9-13900K (2023)
• Microarchitecture: Intel 7 (Raptor Lake)
• Cores: 24
• Threads: 32
• Base clock speed: 3.0 GHz
• Max turbo frequency: 5.8 GHz
• L3 cache: 36 MB
• Integrated graphics: Intel UHD Graphics 770

16
 Evolution of the computer
Timeline of the computer architecture evolution
 2000s & 2010s: Multicore Processors, Multiprocessors and Supercomputers
o Supercomputers becomes more common.
o Superscalar architecture is also introduced, enabling the simultaneous execution of multiple
instructions.

17
 Evolution of the computer
Timeline of the computer architecture evolution
Frontier Supercomputer (2021, USA):
• Processing Power:
• CPUs: 9,472 AMD Epyc 7A53s "Trento" 64-core CPUs
(606,208 cores total)
• GPUs: 37,888 Radeon Instinct MI250X GPUs (8,335,360
cores total)
• Peak performance: 1.102 exaflops (quintillion (1030 )
floating-point operations per second).
• Memory: 9.2 petabytes (1𝑃𝐵 → 106 𝐺𝑏)
• Storage:
• 75 TB/s internal flash storage system
• 700 petabytes
18
 Evolution of the computer
Timeline of the computer architecture evolution
 2000s & 2010s: Multicore Processors, Multiprocessors and Supercomputers
o Introduction of GPUs (Graphics Processing Units) for optimized for graphics tasks and parallel
processing.
o Integrating both CPU and GPU on a single chip to exploit the massive parallelism of GPUs in
non-graphical applications (GPGPU: general-purpose computing on graphics processing), using
CUDA (Nvidia)(Compute Unified Device Architecture)
o NPUs (Neural Processing Units) for AI applications.
o Internet of Things (IoT)

19
 Evolution of the computer
Timeline of the computer architecture evolution
2020s and Beyond: Artificial Intelligence and Specialization:

 Microprocessor architectures specialize further to meet specific needs, particularly in the field
of artificial intelligence (AI). The emergence of processors dedicated to AI, such as Tensor
Processing Units (TPUs), becomes a significant trend.

 Quantum Computing: Early stages of development, with potential to revolutionize certain fields
like cryptography and materials science.

20
 Evolution of the computer
Central theme addressed in the course

1. What are the metrics used to evaluation the architectures?

2. How to increase the performance of the architectures under constraints?

21
 Evolution of the computer
But why?
As a user, you'll know the system's capabilities, strengths, and limits, with a deeper understanding of the
commands you use.

As a programmer, understanding machine characteristics helps you create more efficient programs. For
example, choosing the appropriate data type for a variable can result in significantly faster performance.

As a systems architect or system analyst, your role involves designing and implementing systems to
fulfill an organization's IT requirements. It's crucial to acknowledge that the variations in cost and
capabilities among the selected components can profoundly impact the organization.

22
 Evolution of the computer
Definition
Computer Architecture: Focuses on the structure and behavior of the computer system and refers to the
logical aspects of system implementation as seen by the programmer. Includes elements such as
instruction sets and formats, operation codes, data types, the number and types of registers, addressing
modes, and main memory access methods, as well as various I/O mechanisms.

Software Hardware Architecture

23
 Evolution of the computer
Optimization targets

Software Instruction set Hardware

architecture (ISA)

Programming Microarchitecture
Compiler Transistor
language

24
 Evolution of the computer
Our focus

Memory

• Memory Hierarchy design

ISA

• Instruction-Level parallelism

DATA

• Data-Level parallelism (Multicore/ Multiprocessor)

25