1  2  3  4.... 1  1  1  1  1......... ?

1  2  3  4.... Discrete mathematics

Algorithms
x y( x  y )

x(  | x )
 1 Chapter 3

x 1
x
?

 x 1
x ?
RIZOAN TOUFIQ
ASSISTANT PROFESSOR
DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING
RAJSHAHI UNIVERSITY OF ENGINEERING & TECHNOLOGY
Complexity of Algorithms

Section 3.3
 Section Summary

 Time Complexity
 Worst-Case Complexity
 Algorithmic Paradigms
 Understanding the Complexity of Algorithms
 The Complexity of Algorithms

 Given an algorithm, how efficient is this algorithm for solving

a problem given input of a particular size? To answer this
question, we ask:
– How much time does this algorithm use to solve a problem?
– How much computer memory does this algorithm use to solve a
problem?
 When we analyze the time the algorithm uses to solve the
problem given input of a particular size, we are studying the
time complexity of the algorithm.
 When we analyze the computer memory the algorithm uses
to solve the problem given input of a particular size, we are
studying the space complexity of the algorithm.
 The Complexity of Algorithms

 In this course, we focus on time complexity. The space

complexity of algorithms is studied in later courses.
 We will measure time complexity in terms of the number of
operations an algorithm uses and we will use big-O and big-
Theta notation to estimate the time complexity.
 We can use this analysis to see whether it is practical to use
this algorithm to solve problems with input of a particular
size. We can also compare the efficiency of different
algorithms for solving the same problem.
 We ignore implementation details (including the data
structures used and both the hardware and software
platforms) because it is extremely complicated to consider
them.
 Time Complexity

 To analyze the time complexity of algorithms, we determine the

number of operations, such as comparisons and arithmetic
operations (addition, multiplication, etc.). We can estimate the time
a computer may actually use to solve a problem using the amount
of time required to do basic operations.
 We ignore minor details, such as the “house keeping” aspects of the
algorithm.
 We will focus on the worst-case time complexity of an algorithm.
This provides an upper bound on the number of operations an
algorithm uses to solve a problem with input of a particular size.
 It is usually much more difficult to determine the average case time
complexity of an algorithm. This is the average number of
operations an algorithm uses to solve a problem over all inputs of a
particular size.
 Complexity Analysis of
Algorithms
Example: Describe the time complexity of the algorithm
for finding the maximum element in a finite sequence.

procedure max(a1, a2, …., an: integers)

max := a1
for i := 2 to n
if max < ai then max := ai
return max{max is the largest element}

Solution: Count the number of comparisons.

• The max < ai comparison is made n − 1 times.
• Each time i is incremented, a test is made to see if i ≤ n.
• One last comparison determines that i > n.
• Exactly 2(n − 1) + 1 = 2n − 1 comparisons are made.

Hence, the time complexity of the algorithm is Θ(n).

 Worst-Case Complexity
of Linear Search
Example: Determine the time complexity of the linear
search algorithm.
procedure linear search(x:integer,
a1, a2, …,an: distinct integers)
i := 1
while (i ≤ n and x ≠ ai)
i := i + 1
if i ≤ n then location := i
else location := 0
return location{location is the subscript of the term that equals x, or is 0 if x is
not found}

Solution: Count the number of comparisons.

• At each step two comparisons are made; i ≤ n and x ≠ ai .
• To end the loop, one comparison i ≤ n is made.
• After the loop, one more i ≤ n comparison is made.
If x = ai , 2i + 1 comparisons are used. If x is not on the list, 2n + 1
comparisons are made and then an additional comparison is used to exit
the loop. So, in the worst case 2n + 2 comparisons are made. Hence, the
complexity is Θ(n).
 Average-Case Complexity of
Linear Search
Example: Describe the average case performance of the linear
search algorithm. (Although usually it is very difficult to
determine average-case complexity, it is easy for linear
search.)
Solution: Assume the element is in the list and that the
possible positions are equally likely. By the argument on the
previous slide, if x = ai , the number of comparisons is 2i +
1.

Hence, the average-case complexity of linear search is Θ(n).

 Worst-Case Complexity of
Binary Search
Example: Describe the time complexity of binary search
in terms of the number of comparisons used.

procedure binary search(x: integer, a1,a2,…, an: increasing integers)

i := 1 {i is the left endpoint of interval}
j := n {j is right endpoint of interval}
while i < j
m := ⌊(i + j)/2⌋
if x > am then i := m + 1
else j := m
if x = ai then location := i
else location := 0
return location{location is the subscript i of the term ai equal to x, or 0 if x is not found}

Solution: Assume (for simplicity) n = 2k elements. Note that k = log n.

• Two comparisons are made at each stage; i < j, and x > am .
• At the first iteration the size of the list is 2k and after the first iteration it is 2k-1. Then 2k-2 and so
on until the size of the list is 21 = 2.
• At the last step, a comparison tells us that the size of the list is the size is 20 = 1 and the element is
compared with the single remaining element.
• Hence, at most 2k + 2 = 2 log n + 2 comparisons are made.
• Therefore, the time complexity is Θ (log n), better than linear search.
 Worst-Case Complexity of
Bubble Sort
Example: What is the worst-case complexity of bubble
sort in terms of the number of comparisons made?

procedure bubblesort(a1,…,an: real numbers

with n ≥ 2)
for i := 1 to n− 1
for j := 1 to n − i
if aj >aj+1 then interchange aj and aj+1
{a1,…, an is now in increasing order}
Solution: A sequence of n−1 passes is made through the list. On
each pass n − i comparisons are made.

The worst-case complexity of bubble sort is Θ(n2) since

 Worst-Case Complexity of
Insertion Sort
Example: What is the worst-case complexity of insertion
sort in terms of the number of comparisons made?

procedure insertion sort(a1,…,an:

real numbers with n ≥ 2)
for j := 2 to n
Solution: The total number of i := 1
while aj > ai
comparisons are: i := i + 1
m := aj
for k := 0 to j − i − 1
aj-k := aj-k-1
ai := m
Therefore the complexity is
Θ(n2).
 Matrix Multiplication Algorithm

 The definition for matrix multiplication can be

expressed as an algorithm; C = A B where C is an m
n matrix that is the product of the m k matrix A and
the k n matrix B.
 This algorithm carries out matrix multiplication based
on its definition.
procedure matrix multiplication(A,B: matrices)
for i := 1 to m
for j := 1 to n
cij := 0
for q := 1 to k
cij := cij + aiq bqj
return C{C = [cij] is the product of A and B}
 Complexity of Matrix
Multiplication
Example: How many additions of integers and
multiplications of integers are used by the matrix
multiplication algorithm to multiply two n n matrices.
Solution: There are n2 entries in the product. Finding each
entry requires n multiplications and n − 1 additions. Hence,
n3 multiplications and n2(n − 1) additions are used.
Hence, the complexity of matrix multiplication is O(n3).
 Boolean Product Algorithm

 The definition of Boolean product of zero-one matrices

can also be converted to an algorithm.

procedure Boolean product(A,B: zero-one matrices)

for i := 1 to m
for j := 1 to n
cij := 0
for q := 1 to k
cij := cij ∨ (aiq ∧ bqj)
return C{C = [cij] is the Boolean product of A and B}
 Complexity of Boolean
Product Algorithm
Example: How many bit operations are used to find A
⊙ B, where A and B are n n zero-one matrices?
Solution: There are n2 entries in the A ⊙ B. A total of n Ors
and n ANDs are used to find each entry. Hence, each entry
takes 2n bit operations. A total of 2n3 operations are used.
Therefore the complexity is O(n3)
 Matrix-Chain
Multiplication
 How should the matrix-chain A1A2∙ ∙ ∙An be computed using the
fewest multiplications of integers, where A1 , A2 , ∙ ∙ ∙ , An are m1
m2, m2 m3 , ∙ ∙ ∙ mn mn+1 integer matrices. Matrix
multiplication is associative (exercise in Section 2.6).
Example: In which order should the integer matrices A1A2A3 -
where A1 is 30 20 , A2 20 40, A3 40 10 - be multiplied to use the
least number of multiplications.
Solution: There are two possible ways to compute A1A2A3.
– A1(A2A3): A2A3 takes 20 ∙ 40 ∙ 10 = 8000 multiplications. Then multiplying A1
by the 20 10 matrix A2A3 takes 30 ∙ 20 ∙ 10 = 6000 multiplications. So the
total number is 8000 + 6000 = 14,000.
– (A1A2)A3: A1A2 takes 30 ∙ 20 ∙ 40 = 24,000 multiplications. Then multiplying the
30 40 matrix A1A2 by A3 takes 30 ∙ 40 ∙ 10 = 12,000 multiplications. So the
total number is 24,000 + 12,000 = 36,000.
So the first method is best.

An efficient algorithm for finding the best order for matrix-chain

multiplication can be based on the algorithmic paradigm known as
dynamic programming. (see Ex. 57 in Section 8.1)
 Algorithmic Paradigms

 An algorithmic paradigm is a a general approach based

on a particular concept for constructing algorithms to
solve a variety of problems.
– Greedy algorithms were introduced in Section 3.1.
– We discuss brute-force algorithms in this section.
– We will see divide-and-conquer algorithms (Chapter 8), dynamic
programming (Chapter 8), backtracking (Chapter 11), and
probabilistic algorithms (Chapter 7). There are many other
paradigms that you may see in later courses.
 Brute-Force Algorithms

 A brute-force algorithm is solved in the most

straightforward manner, without taking advantage of
any ideas that can make the algorithm more efficient.
 Brute-force algorithms we have previously seen are
sequential search, bubble sort, and insertion sort.
 Computing the Closest Pair of
Points by Brute-Force
Example: Construct a brute-force algorithm for finding
the closest pair of points in a set of n points in the plane
and provide a worst-case estimate of the number of
arithmetic operations.
Solution: Recall that the distance between (xi,yi) and (xj,
yj) is . A brute-force algorithm
simply computes the distance between all pairs of points
and picks the pair with the smallest distance.

Note: There is no need to compute the square root, since the square of the distance
between two points is smallest when the distance is smallest.

Continued →
 Computing the Closest Pair of
Points by Brute-Force
 Algorithm for finding the closest pair in a set of n points.
procedure closest pair((x1, y1), (x2, y2), … ,(xn, yn): xi, yi real numbers)
min = ∞
for i := 1 to n
for j := 1 to i
if (xj − xi)2 + (yj − yi)2 < min
then min := (xj − xi)2 + (yj − yi)2
closest pair := (xi, yi), (xj, yj)
return closest pair

 The algorithm loops through n(n −1)/2 pairs of points, computes the
value (xj − xi)2 + (yj − yi)2 and compares it with the minimum, etc.
So, the algorithm uses Θ(n2) arithmetic and comparison operations.
 We will develop an algorithm with O(log n) worst-case complexity in
Section 8.3.
Understanding the Complexity
of Algorithms
Understanding the Complexity
of Algorithms

Times of more than 10100 years are indicated

with an *.
 Complexity of Problems

 Tractable Problem: There exists a polynomial time algorithm

to solve this problem. These problems are said to belong to
the Class P.
 Intractable Problem: There does not exist a polynomial time
algorithm to solve this problem
 Unsolvable Problem : No algorithm exists to solve this
problem, e.g., halting problem.
 Class NP: Solution can be checked in polynomial time. But no
polynomial time algorithm has been found for finding a
solution to problems in this class.
 NP Complete Class: If you find a polynomial time algorithm
for one member of the class, it can be used to solve all the
problems in the class.
 P Versus NP Problem

 The P versus NP problem asks whether the class P = NP? Are there
problems whose solutions can be checked in polynomial time, but can
not be solved in polynomial time?
– Note that just because no one has found a polynomial time algorithm is different
from showing that the problem can not be solved by a polynomial time
algorithm.
 If a polynomial time algorithm for any of the problems in the NP
complete class were found, then that algorithm could be used to obtain
a polynomial time algorithm for every problem in the NP complete
class.
– Satisfiability (in Section 1.3) is an NP complete problem.
 It is generally believed that P≠NP since no one has been able to find a
polynomial time algorithm for any of the problems in the NP complete class.
 The problem of P versus NP remains one of the most famous unsolved
problems in mathematics (including theoretical computer science). The Clay
Mathematics Institute has offered a prize of $1,000,000 for a solution.

Stephen Cook
(Born 1939)
 Query???

1  2  3  4.... x y( x  y )  ?


 x 1 x  ?  1
 x 1
x
?

x(  | x )  ? x y( x  y )  ?

 1
1  2  3  4....  ?
1  1  1  1  1......... ?
 x 1
x
?