PERFORMANCE MEASURES

Performance measures
Performance
 Why do we care about performance evaluation?
 Purchasing perspective
◼ given a collection of machines, which has the
◼ best performance ?
◼ least cost ?
◼ best performance / cost ?
 Design perspective
◼ faced with design options, which has the
◼ best performance improvement ?
◼ least cost ?
◼ best performance / cost ?
 How to measure, report, and summarize
performance?
 –Performance metric
 –Benchmark
The environment and its parameters
 Time
 the most fundamental of concepts needed for computer
systems performance analysis.
 Events
 describes an entity of interest in our system
 Measurement
 determine how to measure, when to measure, or what to
measure,
 Intervals
 represents a period of time bounding the initiation of an
event sequence of interest and the end of this event
sequence
 The environment and its parameters
 Response
 the point of issuing some action or command until the
completion and return of control for the requested command.
 Independence
 the occurrence of one does not influence the outcome of the
other
 Randomness
 implies that there is not a pattern that can be mapped onto the
events to determine when they will occur again
 Workloads
 represents how many of some event sequence are being
offered for execution during some given period of time
Which of these airplanes has the
best performance?

 What metric defines performance?

 Capacity, cruising range, or speed?
Two Categories (Notion) of
Performance
 System-oriented measure
 typically revolve around the concepts of
throughput and utilization.
 User-oriented measure
 typically include response time or turnaround
time.
Execution Time
 Elapsed Time
 counts everything (disk and memory accesses, I/O ,
etc.)
 a useful number, but often not good for comparison
purposes
 CPU time
 doesn't count I/O or time spent running other
programs
 can be broken up into system time, and user time
 Our focus: user CPU time
 time spent executing the lines of code that are "in" our
program
Definitions
 Performance is in units of things-per-second
 bigger is better
 If we are primarily concerned with response
time

" X is n times faster than Y" means

Which one is faster?
Concorde or Boeing 747
 Response Time of Concorde vs. Boeing 747?
 Concord is 1350 mph / 610 mph = 2.2 times faster
 Throughput of Concorde vs. Boeing 747 ?
 Boeing is 286,700 pmph / 178,200 pmph= 1.6 “times
faster”

 Boeing is 1.6 times (“60%”) faster in terms of

throughput
 Concord is 2.2 times (“120%”) faster in terms of
flying time
Example of Relative Performance

 If computer A runs a program in 10 seconds

and computer B runs the same program in 15
seconds, how much faster is A than B?
(1) PerformanceA / PerformanceB = n
(2) Performance ratio: 15 /10 = 1.5
(3) A is 1.5 times faster than B

We will focus primarily on execution time for a

single job !
CPU time for a program
 We denote the number of CPU clock cycles for
executing a job to be the cycle count (CC), the
cycle time by CT, and the clock frequency by
f = 1/CT. The time taken by the CPU to
execute a job can be expressed as:
CPU time = CC * CT = CC/f
 Now that we understand cycles
 A given program will require
 some number of instructions (machine instructions)
 some number of cycles

 some number of seconds

 We have a vocabulary that relates these quantities:

 cycle time(seconds per cycle)
 clock rate(cycles per second)

 CPI(cycles per instruction)

◼ a floating point intensive application might have a higher CPI
 MIPS (millions of instructions per second)
◼ this would be higher for a program using simple instructions
Clock Cycles per Instruction (CPI)
Instruction Categories
 ALU (simple assignment and arithmetic and
logic instructions)
 Load
 Store
 Branch
 and so on
 In the case that the CPI for each instruction
category is known, the overall CPI can be
computed as:

 where Ii is the number of times an instruction

of type i is executed in the program and CPIi is
the average number of clock cycles needed to
execute such instruction.
 Example
 Consider computing the overall CPI for a machine A for
which the following performance measures were recorded
when executing a set of benchmark programs. Assume that
the clock rate of the CPU is 200 MHz.
Solution
 Assuming the execution of 100 instructions,
the overall CPI can be computed as:
MIPS (Million Instructions-Per-
Second)
 the rate of instruction execution per unit time,
which is defined as:
 Example
 Suppose that the same set of benchmark programs
considered above were executed on another machine, call
it machine B, for which the following measures were
recorded.

 What is the MIPS rating for the machine considered in the

previous example (machine A) and machine B assuming a
clock rate of 200 MHz?
Solution
Seatwork
Performance Measure: Problem 1

 Consider having a program that runs in 60 s

on computer A, which has a 500 MHz clock.
We would like to run the same program on
another machine, B, in 25 s. If machine B
requires 4.2 times as many clock cycles as
machine A for the same program, what clock
rate must machine B have in MHz?
Performance Measure: Problem 2

 Suppose that we have two implementations of

the same instruction set architecture. Machine
A has a clock cycle time of 50 µs and a CPI of
5.5 for some program, and machine B has a
clock cycle of 65 µs and a CPI of 5.2 for the
same program. Which machine is faster and
by how much?
 Performance Measure: Problem 3
Instruction CPI of the
 A compiler designer is trying to decide
class instruction class
between two code sequences for a
particular machine. The hardware A 4
designers have supplied the B 1
following facts: C 3
 For a particular high-level language, the compiler writer is
considering two sequences that require the following instruction
counts:
Code Instruction counts (in millions)
sequence A B C
1 5 2 4
2 3 5 6

 What is the CPI for each sequence? Which code sequence is faster?
By how much?
 Performance Measure: Problem 4

Consider a machine with three CPI of the

 Instruction class
instruction class
instruction classes and CPI
A 4
measurements as follows: B 3
C 5
 Suppose that we measured the code for a given program in two
different compilers and obtained the following data:
Code Instruction counts (in millions)
sequence A B C
Compiler 1 13 7 6
Compiler 2 23 5 4
 Assume that the machine’s clock rate is 500 MHz. Which code
sequence will execute faster according to MIPS? And according to
execution time?