Divide and Conquer

Divide and Conquer

 Principle
◼ Many algorithms have a recursive structure
 To solve a given problem, the algorithm uses itself to solve
sub-problems with small size, finally combining the results
to obtain the solution

◼ Three steps
 Divide: divide the problem into sub-problems
 Conquer: solve sub-problems recursively, if sub-problems
are small enough then solve directly
 Combination: the results of the sub-problems are the
solutions to the given problem

2
Divide and conquer
 General structure
divide-and-conquer(x: problem): solution
begin
if (x is small and simple) then
return(solution)
else
divide x into many sub-problems x1 , x2 , … xk
for i from 1 to k do
si = divide-and-conquer(xi)
endfor
combine the si solutions into solution s of x
return (s)
endif
end

3
Some applications
 Find the maximum and minimum values
 Multiply two integers
 Multiply two matrices
 Quicksort

4
Find the maximum and minimum values
 Problem
◼ Find the maximum value (max) and minimum value (min) in a
list A[1..n]. How many comparisons are needed between the
elements of A?
 Brute force algorithm
maxmin(A, n)
begin
max = A[1]
min = A[1]
for i from 2 to n
if (A[i] > max) then max = A[i]
endif
if (A[i] < min) then min = A[i]
endif
endfor
return (max, min) 5
end
Find the maximum and minimum values
 Divide and conquer algorithm
◼ Divide the array into two sub-arrays/halves
◼ Conquer sub-solutions
 Apply the idea to each half

◼ Combine the results

 The maximum and minimum values of the array are
obtained by comparing the maximum and minimum
values of two halves

6
Find the maximum and minimum values
 Divide and conquer algorithm

maxmin(A,x,y)
begin
if (y-x  1 ) then
return (max(A[x], A[y])), min(A[x], A[y]))
else
(max1, min1) = maxmin(A, x, (x+y)/2 )
(max2, min2) = maxmin(A, (x+y)/2+1, y )
return (max(max1, max2), min(min1, min2))
endif
end

7
Find the maximum and minimum values
 Algorithm analysis
◼ Calculate the number of comparisons
▪ C(n): number of comparisons, with n = y – x + 1
▪ Suppose n is a power of 2, that means y–x is odd and x+y
is also odd
▪ Size of sub-problems
x + y −1 y − x +1 n
( x + y ) / 2 − x + 1 = − x +1 = =
2 2 2
x + y −1 y − x +1 n
y − ( ( x + y ) / 2 + 1) + 1 = y − = =
2 2 2
▪ So
 1 if n = 2
C ( n) = 
2C (n / 2) + 2 else

8
Find the maximum and minimum values
 Algorithm analysis
◼ Calculate the number of comparisons
C (n) = 2C (n / 2) + 2
= 2 2 C (n / 4) + 2 2 + 2
= 23 C (n / 8) + 23 + 2 2 + 2
log n −1
= 2i C (n / 2i ) + k =1 2 k = 2log n −1 C (2) + 2
i k

k =1
log n −1+1
1 − 2 2 log n
3
= 2log n −1 + = + 2log n − 1 = n − 1
1− 2 2 2

The divide and conquer algorithm uses only 75% of the

comparisons compared to the brute force algorithm
9
Multiply two integers
 Idea
◼ The number of digits of two integers must be equal and
must be equal to the power of 2
 If the number of digits of two integers is different, add zeros to
the left
◼ Idea: we replace the multiplication of two n-digit
integers by 4 multiplications of two n/2-digit integers.
Next, replace the multiplication of two n/2-digit
integers by 4 multiplications of two n/4-digit integers…
◼ We need to multiply two integers: a and b. Let aL and
aR be the left and right halves of the number a,
similarly bL and bR be the left and right halves of b

10
Multiply two integers
 Idea

a * b = (aL * 10n/2 + aR )*(bL * 10n/2 + bR )

= aL * bL * 10n + aL * bR * 10n/2 + aR * bL * 10n/2 + aR * bR
= aL * bL * 10n + (aL * bR + aR * bL )10n/2 + aR * bR

To multiply a and b by the halves of a and b, four

multiplications are required. 11
Multiply two integers
 Design algorithm
◼ Divide: Divide each n-digit integer into two n/2-digit
integers.
◼ Conquer: Perform multiplication of two n/2-digit
integers by recursive calls.
◼ Combination: Perform addition of two integers with
n/2-digits.

12
Multiply two integers
 The divide and conquer algorithm
MultiplyIntegers(a, b, n)
begin
if (n = 1) then
return a × b
else
Divide a into two halves aL and aR , b into two halves bL and bR
p1  MultiplyIntegers(aL , bL , n/2)
p2  MultiplyIntegers(aL , bR , n/2)
p3  MultiplyIntegers(aR , bL , n/2)
p4  MultiplyIntegers(aR , bR , n/2)
return (p1 × 10n + (p2 + p3 ) × 10n/2 + p4 )
end if
end 13
Multiply two integers
 Running time
 c if n = 1
C ( n) = 
4C (n / 2) + dn if n  1
C (n) = 4C (n / 2) + dn
= 42 C (n / 4) + 4d (n / 2) + dn
= 43 C (n / 8) + 22 dn + 2dn + dn
i −1
= 4 C (n / 2 ) + dn 2k
i i

k =0
log n −1
=4 log n
C (1) + dn 
k =0
2k

1 − 2log n −1+1
= cn + dn2

1− 2
= cn 2 + dn 2 − dn. 14
Multiply two integers
 The Karatsuba algorithm
◼ Reduce the number of multiplications of a and b
by the halves of a and b from 4 to 3
Let: p = aL * bL
q = a R * bR
r = (aL + aR ) * (bL + bR)
Then:
a * b = aL * bL * 10n + (aL * bR + aR * bL) * 10n/2 + aR * bR
= p * 10n + (r – p – q) * 10n/2 + q

15
Multiply two integers
 The Karatsuba algorithm
Karatsuba(a, b, n)
begin
if (n = 1) then
return a × b
else
Divide a into two halves aL and aR , b into two halves bL and bR
p  Karatsuba(aL , bL , n/2)
q  Karatsuba(aL , bR , n/2)
r  Karatsuba(aR + aL, bR + bL , n/2)
return (p × 10n + (r – p – q) × 10n/2 + q)
end if
end
16
 C (n) = 3C (n / 2) + dn

Multiply two integers n 3

= 32 C ( ) + dn + dn
4 2
i −1 k
The Karatsuba algorithm n 3
 = 3i C ( i ) + dn  
◼ Running time 2 k =0  2 
log n −1 k
3
c
C ( n) = 
if n = 1 = 3log n C (1) + dn   
2
3C (n / 2) + dn if n  1 k =0
log n −1+1
1 − (3 / 2)
= cnlog3 + dn
1− 3 / 2
= cnlog3 + 2dnnlog3/2 − 2dn
O( n log 3 ) = O( n 1.59 ) = cnlog3 + 2dnnlog3−log 2 − 2dn
nlog3
= cn + 2dn
log3
− 2dn
n
= cnlog3 + 2dnlog3 − 2dn.
17
Multiply two matrices
 Problem
◼ Multiply two n by n matrices: C = A.B
 Brute force algorithm
matrixproduct(A, B, n)
begin
for i from 1 to n
for j from 1 to n
C(i,j) = 0
for k from 1 to n
C(i,j) = C(i,j) + A(i,k) * B(k,j)
endfor
endfor
endfor
return (C)
end
18
Algorithm that performs O(n3) additions and multiplications
Multiply two matrices
 Divide and conquer algorithm (1)
◼ Suppose n = 2k
◼ Divide matrices A, B, C into matrices of size n/2, then
C = A.B respectively

 r s   a b  e f
  =   
 t u   c d  g h
◼ From there, we have
r = ae + bg
s = af + bh
t = ce + dg
u = cf + dh

19
Multiply two matrices
 Divide and conquer algorithm (2)
matrixproduct(A, B, n)
begin
if (n = 1) then return (A.B)
else
Divide A and B into 8 matrices of size n/2: a, b, …, h
r = matrixproduct(a, e, n/2) + matrixproduct(b, g, n/2)
s = matrixproduct(a, f, n/2) + matrixproduct(b, h, n/2)
t = matrixproduct(c, e, n/2) + matrixproduct(d, g, n/2)
u = matrixproduct(c, f, n/2) + matrixproduct(d, h, n/2)
Combining r, s, t and u gives matrix C
return C
end if
end

Complexity: C(n) = 8C(n/2) + n2 , C(1) = 1

where n2 is the number of additions 20
Multiply two matrices
 Divide and conquer algorithm (3)
C (n) = 8C (n / 2) + n 2
= 8(8C (n / 4) + (n / 2) 2 ) + n 2
= 82 C (n / 4) + 2n 2 + n 2
log n −1

k = 0 2 = 8 
i −1
= 8 C (n / 2 ) + n
i i 2 k log n
C (1) + n 2
2 k

k =0
log n −1+1
1 − 2
= 8log n + n 2 = 8log n + n 2 (2log n − 1) =
1− 2
= 23log n + n 2 (n − 1) = n 3 + n 2 (n − 1) = 2n 3 − n 2

O(n3)
21
Multiply two matrices
 The Strassen algorithm (1)
m1 = (a+c)(e+f)
m2 = (b+d)(g+h) + Perform only 7 matrix
m3 = (ad)(e+h) multiplications compared to 8 matrix
m4 = a(fh) multiplications of the divide and
conquer algorithm
m5 = (c+d)e
m6 = (a+b)h + Perform more additions and
m7 = d(ge) subtractions matrix than the divide
◼ Then and conquer algorithm
r = m2 + m3 - m6 - m7
s = m 4 + m6
t = m5 + m7
u = m 1 - m3 - m 4 - m5
22
Multiply two matrices
 The Strassen algorithm (2)
◼ Why correct?
r = m2 + m3 - m6 - m7
= (b+d)(g+h) + (ad)(e+h) - (a+b)h - d(ge)
= bg+bh+dg+dh+ae+ah-de-dh-ah-bh-dg+de
= ae + bg

s = m4 + m6 = a(fh) + (a+b)h = af + bh

t = m5 + m7 = (c+d)e + d(ge) = ce + dg

u = m1 - m3 - m4 - m5
= (a+c)(e+f) - (ad)(e+h) - a(fh) - (c+d)e
= ae+af+ce+cf-ae-ah+de+dh-af+ah-ce-de
= cf + dh

23
Multiply two matrices
 The Strassen algorithm (3)
matrixproduct(A, B, n)
begin
if (n = 1) then return (A.B)
else
Divide A and B into 8 matrices of size n/2: a, b, …, h
m1 = matrixproduct(a+c, e+f, n/2)
m2 = matrixproduct(b+d, g+h, n/2)
m3 = matrixproduct(ad, e+h, n/2)
m4 = matrixproduct(a, fh, n/2)
m5 = matrixproduct(c+d, e, n/2)
m6 = matrixproduct(a+b, h, n/2)
m7 = matrixproduct(d, ge, n/2)
r = m2 + m3 - m6 - m7
s = m4 + m6
t = m5 + m7
u = m1 - m3 - m4 - m5
end if
end 24
Multiply two matrices
 The Strassen algorithm (4)
◼ Complexity
C(n) = 7C(n/2) + 18n2/4
C(1) = 1

25
Multiply two matrices
 The Strassen algorithm (5)
◼ Complexity

9
C (n) = 7C (n / 2) + n 2
2
9 9
= 7(7C (n / 4) + (n / 2) 2 ) + n 2
2 2
9 9
= 7 2 C (n / 8) + 7n 2 / 4 + n 2
2 2
9 2 log n −1  7 
k k
9 2 i −1  7 
= 7 C (n / 2 ) + n k =0   = 7 C (1) + n   
i i log n

2 4 2 k =0  4 

9 2 (7 / 4) log n −1+1 − 1 9 2 ((7 / 4) log n − 1)

=7 log n
+ n =7 + n
log n
=
2 7 / 4 −1 2 3/ 4
 n log 7 
=n log 7
+ 6n ((7 / 4)
2 log n
− 1) = n log 7
+ 6n ( n
2 log 7 − log 4
− 1) = n log 7
+ 6n  2 − 1
2

 n 
= O(n log 7 ) = O(n 2.8 )
26
Quicksort
 Divide and conquer algorithm

quicksort (A)
begin
if (n = 1) then return (A)
else
Select an element x in list A
Split list A into A1 , A2 , A3 such that the elements of A1 is
less than x, the elements of A2 are equal to x, and the
elements of A3 is greater than x
return (quicksort(A1), A2 , quicksort(A3))
end if
end

27
Quicksort
 Worst case complexity
◼ The list A is sorted and we choose x as the first
element

 0 if n  1
C ( n) = 
C (n − 1) + n − 1 else

◼ So: (n2 )

28
Quicksort
 Best case complexity
◼ The selected element x is always the median value, i.e.
split into two sublists of size  n/2, i.e. i=n/2

 0 if n  1
C ( n) = 
2C (n / 2) + n − 1 else
◼ So: O(nlog(n))

29
Quicksort
 Average case complexity (1)

◼ C(n): number of comparisons when sorting list A with n

different elements
◼ C(0) = C(1) = 0

◼ Suppose x is the i-th smallest element in list A

|A1 | = i-1
|A2 | = 1
|A3 | = n-i
◼ Suppose the probability of the ith smallest element x in A
is 1/n
30
Quicksort
 Average case complexity (2)
◼ The recursive call takes average time C(i-1) and C(n-i),
with i  [1..n] with probability 1/n
◼ To divide A into A1 , A2 and A3 requires n-1 comparisons
◼ So, with n  2
C (n) = i =1 (C (i − 1) + C (n − i ) ) + n − 1
1 n
n
◼ But
n n n n −1 n −1 n −1

 (C (i − 1) + C (n − i)) =  C (i − 1) +  C (n − i) =  C (i) +  C (i) = 2 C (i)

i =1 i =1 i =1 i =0 i =0 i =2

◼ So, with n  2
2 n −1
C (n) =  C (i ) + n − 1 (1)
n i =2
31
Quicksort
 Average case complexity (3)
◼ Multiply both sides of (1) by n, with n  2 we have
n −1
nC (n) = 2 C (i ) + n 2 − n (2)
i =2
◼ Replace n by n-1, with n  3 we have
n−2
(n − 1)C (n − 1) = 2 C (i ) + n 2 − 3n + 2 (3)
i =2

◼ Take (2) minus (3)

nC (n) − (n − 1)C (n − 1) = 2C (n − 1) + 2(n − 1)
◼ So
nC (n) = (n + 1)C (n − 1) + 2(n − 1) (4)
◼ Divide both sides of (4) by n(n+1)
C (n) /(n + 1) = C (n − 1) / n + 2(n − 1) / n(n + 1) (532)
Quicksort
 Average case complexity (4)
◼ Let S(n) = C(n)/(n+1), from (5) we have, with n 3

S (n) = S (n − 1) + 2(n − 1) / n(n + 1) (6)

with S(0) = S(1) = 0
(5) is true even when n = 2, S(2)=C(2)/3=1/3
◼ So
0 if n  1
S ( n)  
 S (n − 1) + 2 / n else
S (n)  S (n − 1) + 2 / n  S (n − 2) + 2 /(n − 1) + 2 / n
 S (n − 3) + 2 /(n − 2) + 2 /(n − 1) + 2 / n
n
 S (n − i ) + 2 1 / k
k = n −i +1
33
Quicksort
 Average case complexity (5)
◼ Substitute i = n-1, we have
n n n
1 1 1
S (n)  S (1) + 2 = 2  2 dx = 2 ln n
k =2 k k =2 k 1
x
◼ So
C (n) = (n + 1) S (n)
 2(n + 1) ln n
log n
 2(n + 1)
log e
 1.386(n + 1) log n
◼ So O(nlog(n))
34
Exercises
 Problem 1
◼ Design a divide and conquer algorithm to compute the sum
of an array of numbers.
◼ Evaluate the running time of the algorithm.

 Problem 2
◼ Prove that it is possible to multiply two polynomials ax+b
and cx+d with only 3 multiplications (hint: one of the
multiplications (a+b)(c+d))

35
Exercises
 Problem 3
1. Construct two divide-and-conquer algorithms
3
to multiply
two polynomials with complexity O ( n log 2 )
a. The first algorithm requires splitting the polynomial into two
polynomials, one half of degree n/2 (power [0..n/2]) and the
other half of degree n (power [n/2+1..n])
b. The second algorithm requires splitting the polynomial into two
polynomials, one half with an even exponent, the other half
with an odd exponent

2. Prove that two integers represented by n bits can be

multiplied by an algorithm of complexity O(n log 2 )
3

36
Sum
n
F (n) =  f (i )
For example: f(x)=ip
i =1

( function f increases ) 5
f (t )dt £ å f (i )
5
ò 0
i =1

Overview

n n

ò f (t )dt £ å f (i )
0
i =1
0 1 2 3 4 5
37
 Sum
 Sum (2)

(function f increases) 5

å f (i ) £ ò
6
f (t )dt
1
i =1

Overview
n n +1
å f (i ) £ ò
i =1
1
f (t )dt

0 1 2 3 4 5 6
n n +1
f (t )dt £ å f (i ) £ ò
n
So : ò 0
i =1
1
f (t )dt
n +1 n
If the function f
 f (t )dt   f (i )   f (t )dt
n

decreases: 1 0 38
i =1