Divide-and-Conquer

Reading: CLRS Sections 2.3, 4.1, 4.2, 4.3, 28.2, 33.4.

CSE 780 Analysis of Algorithms

Steve Lai
Divide and Conquer
 Given an instance x of a problem, the divide-and-conquer
method works as follows:

function DAQ(x)
if x is sufficiently small then
solve it directly
else
divide x into smaller subinstances x1 , … , xk ;
for i  1 to k do yi  DAQ(xi );
combine these yi's to obtain a solution y for x;
return(y)
2
Analysis of Divide-and-Conquer
 Typically, x1 , … , xk are of the same size, say  n b  .
 In that case, the time complexity of DAQ, T (n), satisfies
a recurrence:

 c if n  n0
T ( n)  
 kT   n b    f (n) if n  n0
 Where f (n) is the running time of dividing x and
combining yi 's.
 What is c ?
 What is n0 ?
3
Mergesort: Sort an array A[1..n]

 procedure mergesort  A[i.. j ]

// Sort A[i.. j ]//
if i  j then return // base case //
m  (i  j ) 2  

mergesort  A[i..m] 
 divide and conquer
mergesort  A[ m  1.. j ] 
merge  A[i..m], A[m  1.. j ] (A[i..m] 

 Initial call: mergesort  A[1..n] 

4
Analysis of Mergesort
 Let T (n) denote the running time of mergesorting an
array of size n.
 T (n) satisfies the recurrence:

 c if n  1
T ( n)  
 T   n 2    T   n 2    (n) if n  1
 Solving the recurrence yields:
T (n)  (n log n)
 We will learn how to solve such recurrences.

5
Linked-List Version of Mergesort
 function mergesort  i, j 
// Sort A[i.. j ]. Initially, link[ k ] = 0, 1  k  n.//
global A[1..n], link [1..n]
if i  j then return (i ) // base case //
m  (i  j ) 2  

ptr1  mergesort  i, m  

ptr 2  mergesort  m  1, j   divide and conquer

ptr  merge  ptr1, ptr 2  
return ( ptr ) 


6
Solving Recurrences
 Suppose a function T ( n) satisfies the recurrence

 c if n  1
T ( n)  
 3T   n 4    n if n  1
where c is a positive constant.

 Wish to obtain a function g ( n) such that T (n)  ( g (n)).

 Will solve it using various methods: Iteration Method,

Recurrence Tree, Guess and Prove, and Master Method

7
Iteration Method
Assume n is a power of 4. Say, n  4 m. Then,
T (n)  n  3T (n / 4)
= n  3 n / 4  3T ( n / 16) 
= n  3( n / 4)  9  ( n / 16)  3T ( n / 64) 
= n  (3 / 4) n  (3 / 4) 2 n  33 T ( n / 43 )
 3 3
2
3
m 1
 m  n 
= n 1            3 T m 
 4 4 4  4 
= n (1)  O( n)  ( n )
So, T (n)  (n | n a power of 4)  T (n)  (n). (Why?)
8
Remark
 We have applied Theorem 7 to conclude T (n)  (n)
from T ( n)  ( n | n a power of 4).

 In order to apply Theorem 7, T ( n) needs to be

nondecreasing.

 It will be a homework question for you to prove that

T (n) is indeed nondecreasing.

9
Recurrence Tree
solving problems time needed
1 of size n
  n
3 of size n / 4
  3 n / 4
32 of size n / 42
  32  n / 42
33 of size n / 43
  33  n / 43
 
3m1 of size n / 4m1 3m1  n / 4m 1
3m of size n / 4m 3m  (1)
10
Guess and Prove
 c if n  1
 Solve T (n)  
 3T   n 4    (n) if n  1
 First, guess T ( n)  ( n), and then try to prove it.
 Sufficient to consider n  4 m , m  0, 1, 2, . . .
 Need to prove: c1 4m  T (4m )  c2 4 m for some c1 , c2
and all m  0. We proceed by induction on m.
 IB: When m  0, c1 40  T (40 )  c2 40 if c1  c  c2 .
 IH: Assume c1 4m 1  T (4m 1 )  c2 4 m 1 for some c1, c2 .

11
 IS: T (4m ) = 3T (4 m 1 )  (4 m )
 3c2 4 m 1  c2 4 m for some constant c2
  3c2 4  c2  4 m
 c2 4 m if c2  c2 4

T (4 m ) = 3T (4 m1 )  (4 m )
 3c1 4 m 1  c14 m for some constant c1
  1
3c 4  c1
 4 m

 c1 4 m if c1  c1 4
 Let c1 , c2 be such that c1  c  c2 , c2  c2 4, c1  c1 4 .
Then, c1 4m  T (4 m )  c2 4m for all m  0.

12
Master's Theorem
 Definition: f (n) is polynomially smaller than g ( n), denoted
as f ( n) g (n), iff f ( n)  O( g ( n) n  ), or f ( n) n  O( g ( n)),
for some   0.

 For example, 1  n n 0.99 n n 2 .

 Is 1 log n ? Or n n log n ?

 To answer these, ask yourself whether or not n  O(log n).

 For convenience, write f ( n)  g ( n) iff f ( n)    g ( n)  .

13
Master's Theorem
If T (n) satisfies the recurrence T ( n)  aT ( n / b)  f ( n),
then T (n) is bounded asymptotically as follows.

1. If f (n) n logb a , then T ( n)   nlogb a . 
2. If f (n) n logb a , then T (n)    f (n)  .
3. If f (n)  nlogb a , then T (n)    f (n)log n  .
4. If f (n)  nlogb a log k n, then T (n)    f (n)log n  .

In case 2, it is required that af ( n / b)  cf ( n) for some c  1,

which is satisfied by most f ( n) that we shall encounter.

In the theorem, n / b should be interpreted as  n / b  or  n / b  .

14
Examples: solve these recurrences
 T (n) = 3T (n / 4)  n.
 T (n)  9T (n / 3)  n.
 T (n)  T (2n / 3)  1.
 T (n)  3T ( n / 4)  n log n.

 T (n)  7T (n / 2)  (n 2 ).
 T (n)  2T (n / 2)  n log n.
 T (n)  T ( n / 3)  T (2n / 3)  n.

15
When recurrences involve roots

 Solve T (n)  
 
 2T n  log n if n  2

 c otherwise
 Suffices to consider only powers of 2. Let n  2m.
 Define a new function S ( m)  T (2 m ).
 The above recurrence translates to
 2S  m / 2   m if m  1
S ( m)  
c otherwise
 By Master Theorem, S ( m)  ( m log m).
 So, T (n)  (log n log log n)

16
Strassen's Algorithm for Matrix Multiplication
 Problem: Compute C  AB, given n  n matrices A and B.
 The straightforward method requires ( n3 ) time,
n
using the formula cij   aik bkj .
k 1

 Toward the end of 1960s, Strassen showed how to multiply

matrices in O( nlog 7 )  O( n 2.81 ) time.
 For n  100, n 2.81  416,869 n3  1,000,000.
 The time complexity was reduced to O( n 2.521813 ) in 1979,
to O(n 2.521801 ) in 1980, and to O( n 2.376 ) in 1986.
 In the following discussion, assume n to be a power of 2.
17
 Write
 A11 A12   B11 B12   C11 C12 
A , B   , C   
 A21 A22   B21 B22  C
 21 C 22 

where each Aij , Bij , Cij is a n / 2  n / 2 matrix.

 C11 C12   A11 A12   B11 B12 

 Then    .
 C21 C22   A21 A22   B21 B22 

 If we compute Cij  Ai1B1 j  Ai 2 B2 j , the running time T ( n)

will satisfy the recurrence T ( n)  8T ( n / 2)  (n 2 )
 T (n) will be ( n3 ), not better than the straightforward one.

18
 Strassen showed

 C11 C12   M2  M3 M1  M 2  M 5  M 6 
  
 C21 C22   M 1  M 2  M 4  M 7 M1  M 2  M 4  M 5 

where M 1  ( A21  A22  A11 )  ( B22  B12  B11 )

M 2  A11  B11
M 3  A12  B21
M 4  ( A11  A21 )  ( B22  B12 )
M 5  ( A21  A22 )  ( B12  B11 )
M 6  ( A12  A21  A11  A22 )  B22
M 7  A22  ( B11  B22  B12  B21 )

 T (n)  7T ( n / 2)  (n 2 )  T (n)  (n log 7 )

19
The Closest Pair Problem
 Problem Statement: Given a set of n points in the plane,
A   ( xi , yi ) : 1  i  n , find two points in A whose
distance is smallest among all pairs.

 Straightforward method: ( n 2 ).
 Divide and conquer: O ( n log n ).

20
The Divide-and-Conquer Approach
1. Partition A into two sets: A  B  C.
2. Find a closest pair ( p1, q1 ) in B.
3. Find a closest pair (p2 , q2 ) in C.
4. Let   min  dist( p1 , q1 ), dist( p2 , q2 ) .
5. Find a closest pair (p3 , q3 ) between B and C with
distance less than  , if such a pair exists.
6. Return the pair of the three which is closest.
 Question : What would be the running time?
 Desired: T (n)  2T (n / 2)  O(n)  T (n)  O(n log n).

21
 Now let's see how to implement each step.
 Trivial: steps 2, 3, 4, 6.

22
Step 1: Partition A into two sets: A  B  C.

 The partition should be approximately even. Why?

 A natural choice is to draw a vertical line to divide
the points into two groups.
 So, sort A[1..n ] by x -coordinate. (Do this only once.)
 Then, we can easily partition any set A[i.. j ] by
A[i.. j ]  A[i..m]  A[m  1.. j ]
where m  (i  j ) 2  .

23
Step 5: Find a closest pair between B and C with
distance less than  , if exists.

 We will write a procedure

Closest-Pair-Between-Two-Sets  A[i.. j ], ptr ,  , ( p3, q3) 

which finds a closest pair between A[i..m] and A[ m  1.. j ]

with distance less than  , if exists.
 The running time of this procedure must be no more than
O  A[i.. j ]  .

24
Data Structures
 Let the coordinates of the n points be stored in X [1..n]
and Y [1..n].

 For simplicity, let A[i]  ( X [i], Y [i]).

 For convenience, introduce two dummy points:
A[0]  (, ) and A[ n  1]  (, )

 We will use these two points to indicate "no pair"

or "no pair closer than  ."

 Introduce an array Link[1..n], initialized to all 0's.

25
Main Program
 Global variable: A[0..n  1]

 Sort A[1..n] such that X [1]  X [2]      X [ n].

That is, sort the given n points by x-coordinate.

 Call Procedure Closest-Pair with appropriate parameters.

26
Procedure Closest-Pair  A[i.. j], ( p, q)  //Version 1//
{*returns a closest pair ( p, q) in A[i.. j ]*}
 If j  i  0 : ( p, q)  (0, n  1);
 If j  i  1: ( p, q)  (i, j );
 If j  i  1: m  (i  j ) 2 
Closest-Pair  A[i..m], ( p1 , q1 ) 
Closest-Pair  A[ m  1.. j ], ( p2 , q2 ) 
ptr  Sort A[i.. j ] by y -coordinate into a linked list
  min  dist( p1, q1 ), dist( p2 , q2 )
Closest-Pair-Between-Two-Sets  A[i.. j ], ptr ,  , ( p3 , q3 ) 
( p, q)  closest of the three ( p1, q1), ( p2 , q2 ), ( p3 , q3 )
27
Time Complexity of version 1
 Initial call: Closest-Pair  A[1..n], ( p, q)  .

 Assume Closest-Pair-Between-Two-Sets needs O(n) time.

 Let T (n) denote the worst-case running time of
Closest-Pair  A[1..n], ( p, q)  .

 Then, T ( n)  2T ( n / 2)  ( n log n).

 So, T (n)    n log 2 n  .

 Not as good as desired.

28
How to reduce the time complexity to O(log n)?
 Observe that when j  i  1, Closest-Pair  A[i.. j ], ( p, q) 
not only computes a closest pair ( p, q), but also sorts
A[i.. j ] by y -ccordinate.
 Now, suppose we implement the statement
ptr  Sort A[i.. j ] by y -coordinate into a linked list
by the linked-list version of Mergesort.
 Then we have version 2 of Procedure Closest-Pair.

29
Procedure Closest-Pair  A[i.. j], ( p, q)  //Version 2//
 If j  i  0 : ( p, q)  (0, n  1);
 If j  i  1: ( p, q)  (i, j );
 If j  i  1: m  (i  j ) 2 
Closest-Pair  A[i..m], ( p1, q1 ) 
Closest-Pair  A[ m  1.. j ], ( p2 , q2 ) 
ptr1  Mergesort  A[i..m] 
ptr 2  Mergesort  A[ m  1.. j ] 
ptr  Merge  ptr1, ptr 2 
(the rest is the same as in version 1)

30
Observations on Version 2
 Since Closest-Pair  A[i..m], ( p1, q1 )  has already sorted
A[i..m], Mergesort  A[i..m] is redundant.

 Similarly, Mergesort  A[ m  1.. j ] is redundant.

 So, we may
 remove the two Mergesorts
 modify Closest-Pair  A[i..m], ( p1, q1 ), ptr1
 modify Closest-Pair  A[m  1.. j ], ( p2 , q2 ), ptr 2 
 add sorting to the two base cases.

 Then we have the final version of Closest-Pair.

31
Procedure Closest-Pair  A[i.. j], ( p, q), ptr  //final version//
{*sort A[i.. j ] by y and return a closest pair ( p, q ) in A[i.. j ]*}
 if j  i  0 : ( p, q)  (0, n  1); ptr  i
 if j  i  1: ( p, q)  (i, j );
if Y [i]  Y [ j ] then  ptr  i; Link[i ]  j
else  ptr  j; Link[ j ]  i
 if j  i  1: m  (i  j ) 2 
Closest-Pair  A[i..m], ( p1 , q1 ), ptr1
Closest-Pair  A[ m  1.. j ], ( p2 , q2 ), ptr 2 
ptr  Merge  ptr1, ptr 2 
(the rest is the same as in version 1)
32
Time Complexity of the final version
 Initial call: Closest-Pair  A[1..n], ( p, q), pqr  .

 Assume Closest-Pair-Between-Two-Sets needs (n) time.

 Let T (n) denote the worst-case running time of

Closest-Pair  A[1..n], ( p, q), pqr  .

 Then, T (n)  2T (n / 2)  ( n).

 So, T ( n)    n log n  .

 Now, it remains to write the procedure

Closest-Pair-Between-Two-Sets  A[i.. j], ptr,  , ( p3 , q3 ) 

33
Closest-Pair-Between-Two-Sets
 Input:  A[i.. j ], ptr ,  
 Output: a closest pair ( p, q) between B  A[i..m] and
C  A[ m  1.. j ] with distance <  , where m  (i  j ) / 2  .
If there is no such a pair, return the dummy pair (0, n  1).
 Time complexity desired : O  A[i.. j ]  .
 For each point b  B, we can only compute dist(b, c) for O (1)
points c  C. Similarly for each point c  C .
 Recall that A[i.. j ] has been sorted by y. We will follow the
sorted linked list and look at each point.

34
Closest-Pair-Between-Two-Sets
 L0 : vertical line passing through the point A[m].
 L1 and L2 : vertical lines to the left and right of L0 by  .
 We observe that:
 We only need to consider those points between L1 and L2 .
 For each point k in A[i..m], we only need to consider the
points in A[ m  1.. j ] that are inside the square of    .
 There are at most three such points.
 And they are among the most recently visited three points
of A[ m  1.. j ] lying between L0 and L2 .
 Similar argument for each point k in A[m  1.. j ].
35
 L1 L0 L2

Red points are apart

by at least δ
k
δ

Blue points are apart For k, only need to consider the

by at least δ points in the square less the
right and bottom edges

36
Closest-Pair-Between-Two-Sets(A[i.. j ], ptr ,  , ( p3 , q3 ))
// Find the closest pair between A[i..m] and A[m  1.. j ]
with dist <  . If there exists no such a pair, then return
the dummy pair (0, n  1). //
global X [0..n  1], Y [0..n  1], Link[1..n]

( p3 , q3 )  (0, n  1)
b1 , b2 , b3  0 //most recently visited 3 points btwn L0 , L1 //
c1 , c2 , c3  n  1 //such points between L0 , L2 //
m  (i  j ) 2 
k  ptr

37
while k  0 do //follow the linked list until end//
1. if X [ k ]  X [ m]   then // consider only btwn L1 , L2 //
if k  m then //point k is to the left of L0 //
compute d  min{dist( k , ci ) : 1  i  3};
if d   then update  and ( p3 , q3 );
b3  b2 ; b2  b1; b1  k ;
else //point k is to the right of L0 //
compute d  min{dist(k , bi ) : 1  i  3};
if d   then update  and ( p3 , q3 );
c3  c2 ; c2  c1; c1  k ;
2. k  Link[k ]
38
Convex Hull
 Problem Statement: Given a set of n points in the plane,
say, A   p1, p2 , p3 ,  , pn  ,
we want to find the convex hull of A.
 The convex hull of A, denoted by CH ( A), is the smallest
convex polygon that encloses all points of A.
 Observation: segment pi p j is an edge of CH ( A) if all

other points of A are on the same side of pi p j (or on pi p j ).
 Straightforward method: ( n 2 ).
 Divide and conquer: O ( n log n).

39
Divide-and-Conquer for Convex Hull
0. Assume all x-coordinates of the given points are distinct.
1. Let A be sorted by x-coordinate.
2. If A  3, solve the problem directly. Otherwise, apply
divide-and-conquer as follows.
3. Break up A into A  B  C.
4. Find the convex hull of B.
5. Find the convex hull of C.
6. Combine the two convex hulls by finding the upper and
lower bridges to connect the two convex hulls.

40
Upper and Lower Bridges
 The upper bridge between CH ( B) and CH (C ) is the
the edge vw, where v  CH ( B) and w  CH (C ), such that

 all other vertices in CH ( B) and CH ( C ) are below vw, or
 the two neighbors of v in CH ( B ) and the two neighbors

of w in CH (C ) are below vw, or
 the counterclockwise-neighbor of v in CH ( B ) and the

clockwise-neighbor of w in CH (C ) are below vw, if we
choose v and w as we will do in the next page.
 Lower bridge: similar.

41
Finding the upper bridge
 v  the rightmost point in CH ( B);
w  the leftmost point in CH (C ).
 Loop

if counterclockwise-neighbor(v) lies above line vw then
v  counterclockwise-neighbor(v )

else if clockwise-neighbor(w) lies above vw then
w  clockwise neighbor(w)
else exit from the loop
 vw is the upper bridge.

42
Data Structure and Time Complexity
 What data structure will you use to represent
a convex hull?
 Using your data structure, how much time will it take
to find the upper and lower bridges?
 What is the over all running time of the algorithm?

 What if we remove the assumption that no two points in A

share a common x-coordinate?

43
Orientation of three points
 Three points: p1  x1 , y1  , p2  x2 , y2  , p3  x3 , y3  .

  p1 , p2 , p3  in that order is counterclockwise if

x1 y1 1
x2 y2 1  0
x3 y3 1

 Clockwise if the determinant is negative.

 Colinear if the determinant is zero.

44