Scribd
Upload
93 views

(2010-02-27) Measuring Performance

This document discusses various methods for measuring computer performance, including benchmarks. It describes common metrics like instructions per cycle (IPC), millions of instructions per second (MIPS), and millions of floating-point operations per second (MFLOPS). It also discusses different types of benchmarks, including program kernels, toy programs, synthetic benchmarks, and the SPEC benchmark suites which use real-world programs modified to minimize I/O effects. The SPEC benchmarks provide two metrics - SPECratio measures how many times faster a system can perform a single task compared to a reference machine, while SPECrate measures throughput by calculating how many benchmark tasks can be completed within a time interval.

Uploaded by

i_2loveu3235
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
93 views

(2010-02-27) Measuring Performance

This document discusses various methods for measuring computer performance, including benchmarks. It describes common metrics like instructions per cycle (IPC), millions of instructions per second (MIPS), and millions of floating-point operations per second (MFLOPS). It also discusses different types of benchmarks, including program kernels, toy programs, synthetic benchmarks, and the SPEC benchmark suites which use real-world programs modified to minimize I/O effects. The SPEC benchmarks provide two metrics - SPECratio measures how many times faster a system can perform a single task compared to a reference machine, while SPECrate measures throughput by calculating how many benchmark tasks can be completed within a time interval.

Uploaded by

i_2loveu3235
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 11

Measuring performance

Kosarev Nikolay
MIPT
Feb, 2010

Agenda
Performance measures
Benchmarks
Summarizing results

Performance measures
Time to perform an individual operation
The first metric. Used if most instructions take the same
execution time.

Instruction mix
Idea is to categorize all instructions into classes by cycles
required to execute an instruction. Average instruction
execution time is calculated (IPC if measured in cycles).
Gibson instruction mix [1970]. Proposed weights for a set of
predefined instruction classes (based on programs running on
IBM 704 and 650)
Depends on the program executed, instruction set. Could be
optimized by compiler. Ignores major performance impacts
(memory hierarchy etc.)

Performance measures (cont.)


MIPS (millions of instructions per second)
Depends on instruction set (the heart of the differences between
RISC and CISC).

Relative MIPS. DEC VAX-11/780 (1 MIPS computer, reference


machine). Relative MIPS of machine M for predefined benchmark:

MFLOPS

(millions of floating-point operations per second)

Metric for supercomputers, tries but not corrects the primary


MIPS shortcoming

Performance measures (cont.)


Execution time
Ultimate measure of performance for a given application, consistent
across systems.
Total execution time (elapsed time). Includes system-overhead
effects (I/O operation, memory paging, time-sharing load, etc).
CPU time. Time spent for execution of application only by
microprocessor.
Better to report both measures for the end user.

Benchmarks
Program kernels
Small programs extracted from real applications. E.g. Livermore
Fortran Kernels (LFK) [1986].
Dont stress memory hierarchy in a realistic fashion, ignore operating
system.

Toy programs
Real applications but too small to characterize programs that are
likely to be executed by the users of a system. E.g. quicksort.

Synthetic benchmarks
Artificial programs, try to match profile and behavior of real
application. E.g. Whetstone [1976], Dhrystone [1984].
Ignore interactions between instructions (due to new ordering) that
lead to pipeline stalls, change of memory locality.

Benchmarks (cont.)
SPEC
SPEC (Standard Performance Evaluation Corporation)
Benchmark suites consist of real programs modified to be portable
and to minimize the effect of I/O activities on performance
5 SPEC generations: SPEC89, SPEC92, SPEC95, SPEC2000 and
SPEC2006 (used to measure desktop and server CPU performance)
Benchmarks organized in two suites: CINT and CFP
2 derived metrics: SPECratio and SPECrate
SPECSFS, SPECWeb (file server and web server benchmarks)
measure performance of I/O activities (from disk or network traffic)
as well as the CPU

Benchmarks (cont.)

Benchmarks (cont.)
SPECratio is a speed metric
How fast a computer can complete single task
Execution time normalized to a reference computer. Formula:

It measures how many times faster than a reference machine one


system can perform a task
Reference machine used for SPEC CPU2000/SPEC CPU2006 is Sun
UltraSPARC II system at 296MHz
Choice of the reference computer is irrelevant in performance
comparisons.

Benchmarks (cont.)
SPECrate is a throughput metric
Measures how many tasks the system completes within an
arbitrary time interval
Measured elapsed time from when all copies of one
benchmark are launched simultaneously until the last copy
finishes
Each benchmark measured independently
User is free to choose # of benchmark copies to run in order
to maximize performance
Formula
Reference factor normalization factor; benchmark duration is normalized to standard job length
(benchmark with the longest SPEC reference time). Unit time used to convert to unit of time more
appropriate for work (e.g. week)

10

You might also like