What is Algorithm Analysis?

Algorithm Analysis How good is the algorithm?

• time efficiency
Lecture 2: The Analysis Framework • space efficiency
Ο, Ω, Θ notations
What’s more important than speed?
• correctness
• robustness
• modularity/maintainability
• security
Fall 2022
Dr. Lin Markowsky

Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 2

Analysis of Algorithms Theoretical Analysis of Time Efficiency

Issues: Time efficiency is analyzed by determining the number of
repetitions of a basic operation as a function of the input size
• correctness
• time efficiency Basic operation: the operation that contributes most towards
• space efficiency the running time of the algorithm
• optimality
input size

Approaches:
T(n) ≈ copC(n)
• theoretical analysis
• empirical analysis running time Number of times
execution time
for basic operation basic operation is
executed

Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 3 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 4
Example: Sequential Search Example: Sequential Search

cost of basic
operation
running time count of basic
operation
• What does this function do?
• We’d like to analyze its running time: T(n) ≈ copC(n)
• What are the valid inputs?
• What is the “input size” for this algorithm? n
• What is returned?
• What is the “basic operation”? key comparison
Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 5 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 6

Example: Element Uniqueness Problem Example: Maximum Element

• What is the “input size” for this algorithm? n • What is the “input size” for this algorithm? n
• What is the “basic operation”? comparison • What is the “basic operation”? comparison
– Why not assignment?
Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 7 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 8
Example: Insertion Sort Example: Matrix Multiplication

• What is the “input size” for this algorithm? n

• What is the “basic operation”? comparison

Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 9 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 10

Example: Matrix Multiplication Example: Counting Binary Digits

• What is the “input size” for this algorithm? n • What is the “input size” for this algorithm? n
• What is the “basic operation”? multiplication • What is the “basic operation”? comparison
– Why not addition? – or the operations inside the while loop

Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 11 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 12
Example: Closest Pair of Points Example: String Matching

• What is the “input size” for this algorithm? n

• What is the “basic operation”? square root • What is the “input size” for this algorithm? (n, m)
– How would you improve this algorithm? • What is the “basic operation”? character comparison

Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 13 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 14

Input Size and Basic Operations Example: Sequential Search

Problem Input Size Basic Operation

searching for a key in a

n key comparison
list of n items

sorting a list of n items comparing items

n
(by comparison) in the list
cost of basic
adding two addition of operation
mxn
m x n matrices two numbers running time count of basic
operation
multiplying two n multiplication of
n x n matrices (or n x n) two numbers • We’d like to analyze its running time: T(n) ≈ copC(n)
visiting a vertex or • How many times is the key comparison performed?
typical graph problem #vertices and/or #edges
traversing an edge – Sometimes the answer depends on the input itself!
Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 15 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 16
Example: Sequential Search Example: Sequential Search

Average-case count of key comparisons (the basic op):

• p = the probability of the key actually being in the list
• Then:
Cave(n) = p ˙ Ckey-is-in-list(n) + (1 – p) ˙ Ckey-not-in-list(n)

• If the key is not in the list, we have to compare the key to every item
in the list, so:
Ckey-not-in-list(n) = n

• If the key is in the list, let’s assume that it’s equally likely to be in any
Worst-case count: Cworst(n) = n position, so:
Best-case count: Cbest(n) = 1 Ckey-is-in-list(n) = (1 + 2 + 3 + ⋯ + i + ⋯ + n) / n

Average-case count: Cave(n) = ? Why? If the key is in the ith position, we have to make i comparisons.

Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 17 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 18

Example: Sequential Search Example: Sequential Search

Worst-case running time:
Average-case count of key comparisons (the basic op):
• Putting it all together: • Tworst(n) ≈ copCworst(n) = cop˙ n
Best-case running time:
(1+2+ 3+⋯+i +⋯+ n)
C ave (n)= p⋅ +(1−p)⋅n • Tbest(n) ≈ copCbest(n) = cop˙ 1
n

p n (n+1)
Average-case running time:
C ave (n)= ⋅ +(1− p)⋅n • Tave(n) ≈ copCave(n) ≈ cop˙ n/2 (if the key is definitely in the list)
n 2
– running time for some “typical input”
p( n+1)
C ave (n)= +(1− p) n – not the average of the worst and best cases
2
– based on the “expected” number of basic operations
• Does this make sense?
– we had to make some assumptions
• How many key comparisons if the key is not in the list?
• About how many key comparisons, on average, if the key is
– probability of a successful search
definitely in the list? – probability of the location of the key if it’s in the list

Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 19 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 20
How does the running time change … ? How does the running time change … ?

Consider the running time for some algorithm: Consider the worst-case running time for sequential search:
T (n)≈c op C (n) T (n)≈c op C worst (n)=c op⋅n
If we run the algorithm on a computer that’s 2x as fast, If input doubles to 2n, how does the running time change?
how does the running time change? T (2 n) c op C (2 n) c op⋅2 n
c op ≈ = =2
⋅C (n) T (n) c op C (n) c op⋅n
T fast (n) 2 1
T (n)
≈ =
c op⋅C (n) 2 If input doubles again, to 4n, how does the running time change?
If we run the algorithm on a computer that’s 4x as fast, T (4 n) c op C (4 n) c op⋅4 n
≈ = =4
how does the running time change? T (n) c op C (n) c op⋅n
c op
⋅C (n) If input doubles again, to 8n, how does the running time change?
T faster (n) 4 1
≈ = T (8 n) c op C (8 n) c op⋅8 n
T (n) c op⋅C (n) 4 ≈ = =8
T (n) c op C (n) c op⋅n
What happened to C(n)? Why is that significant?
What happened to c op? Why is that significant?
Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 21 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 22

How does the running time change … ? Running Time Analysis

Suppose the running time of an algorithm is: We are primarily concerned about the running time of
T (n)≈c op C (n)=c op⋅n 2 algorithms.
If input doubles to 2n, how does the running time change? • To compare different algorithms that solve the same
T (2 n) c op C (2 n) c op⋅(2 n)
2 problem.
≈ = =4
T (n) c op C (n) c op⋅n 2 • To determine how the running time increases with the
If input doubles again, to 4n, how does the running time change? size of the problem.
T (4 n) c op C (4 n) c op⋅(4 n)
2
For some algorithms, the analysis may depend on the
≈ = =16
T (n) c op C (n) c op⋅n
2
input itself:
If input doubles again, to 8n, how does the running time change? • Usually interested in worst-case running time.
T (8 n) c op C (8 n) c op⋅(8 n)
2 • Sometimes interested in average-case running time
≈ = =64 (quicksort).
T (n) c op C (n) c op⋅n2
How does this algorithm compare with sequential search?
• Sometimes interested in best-case running time
(insertion sort).
Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 23
Running Time Analysis Asymptotic Notation
Not concerned about:
• the execution time on a particular machine
• a specific count of operations
Better:
• compare the asymptotic running time as a function of
the input size
• the asymptotic rate of growth
• f(n) = Θ(g(n)): f(n) is asymptotically “=” g(n)
– g(n) is an asymptotically tight bound for f(n)
• f(n) = Ο(g(n)): f(n) is asymptotically “≤” g(n)
– g(n) is an asymptotic upper bound for f(n)
• f(n) = Ω(g(n)): f(n) is asymptotically “≥” g(n)
– g(n) is an asymptotic lower bound for f(n)
Image and ref: Cormen et al., Introduction to Algorithms, 3 rd Edition, page 45.

Comparison of Values as n   Asymptotic Efficiency Classes

1 constant
log n logarithmic
n linear
n log n “n log n”
n2 quadratic
n3 cubic
2n exponential
n! factorial

Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Based on: A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012
Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 27 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. (Modified by L. Markowsky) 28
For next time ...

Review Levitin, Sections 2.1-2.2.

Read Levitin, Section 2.3.