Design and Analysis of Algorithms

Divide and Conquer Algorithms

Edited by
Dr/ Hossam Hawash
Basic algorithmic techniques

When faced with a new algorithmic problem, one should consider

applying one of the following approaches:
 Divide-and-conquer :: divide the problem into two subproblems,
solve each problem separately and merge the solutions
 Dynamic programming :: express the solution of the original
problem as a recursion on solutions of similar smaller problems. Then
instead of solving only the original problem, solve all sub-problems
that can occur when the recursion is unravelled, and combine their
solutions
 Greedy approach :: build the solution of an optimization problem
one piece at a time, optimizing each piece separately
 Inductive approach :: express the solution of the original problem
based on the solution of the same problem with one fewer item; a
special case of dynamic programming and similar to the greedy
approach
The divide-and-conquer strategy

The divide-and-conquer strategy solves a problem by:

1. Breaking it into subproblems (smaller instances of the same problem)
2. Recursively solving these subproblems
[Base case: If the subproblems are small enough, just solve them by
brute force.]
3. Appropriately combining their answers.

Where is the work done?

In three places:
1. In dividing the problems into subproblems.
2. At the tail end of the recursion, when the subproblems are so small
they are solved outright.
3. In the gluing together of the intermediate answers.
Merge sort [CLRS 2.3.1]

Merge sort is a divide-and-conquer algorithm.

Informal description:
It sorts a subarray A[p . . r) := A[p . . r − 1]
Divide by splitting it into subarrays A[p . . q) and A[q . . r) where
q = ⌊(p + r)/2⌋.
Conquer by recursively sorting the subarrays.
Recursion stops when the subarray contains only one element.
Combine by merging the sorted subarrays A[p . . q) and A[q . . r) into a
single sorted array, using a procedure called M ERGE(A, p, q, r).
M ERGE compares the two smallest elements of the two subarrays and
copies the smaller one into the output array.
This procedure is repeated until all the elements in the two subarrays have
been copied.
Example

6 4 8 9 2 1 3
divide ✟✟
✟
✙
❍❍
❥
❍
6 4 8 9 2 1 3
divide ✄ ❅
❘
❅ ✠ ❅
❘
❅
✄ 4 8 9 2 1 3
✄
divide ✄✎ ☛✁ ❆❯ ☛✁ ❆❯ ☛✁ ❆❯
base case 6 4 8 9 2 1 3
merge ❈ ❆❯ ☛✁ ❆❯ ☛✁ ❆❯ ☛✁
❈ 4 8 2 9 1 3
❈
merge ❈❲ ✠ ❅
❘
❅ ✠
4 6 8 1 2 3 9
❍❍ ✟✟
merge ❥
❍ ✟
✙
1 2 3 4 6 8 9
Pseudocode for M ERGE -S ORT

M ERGE -S ORT(A, p, r)

Input: An integer array A with indices p < r.

Output: The subarray A[p . . r) is sorted in non-decreasing order.
1 if r > p + 1
2 q = ⌊(p + r)/2⌋
3 M ERGE -S ORT(A, p, q)
4 M ERGE -S ORT(A, q, r)
5 M ERGE(A, p, q, r)

Initial call: M ERGE -S ORT(A, 1, n + 1)

Merge

Input: Array A with indices p, q, r such that

 p<q<r
 Subarrays A[p . . q) and A[q . . r) are both sorted.
Output: The two sorted subarrays are merged into a single sorted
subarray in A[p . . r).
Pseudocode for M ERGE

M ERGE(A, p, q, r)

1 n1 = q − p 11 i = 1
2 n2 = r − q 12 j = 1
3 Create array L of size n1 + 1 13 for k = p to r − 1
4 Create array R of size n2 + 1 14 if L[i] ≤ R[j]
5 for i = 1 to n1 15 A[k] = L[i]
6 L[i] = A[p + i − 1] 16 i = i+1
7 for j = 1 to n2 17 else A[k] = R[j]
8 R[j] = A[q + j − 1] 18 j = j+1
9 L[n1 + 1] = ∞
10 R[n2 + 1] = ∞
Running time of M ERGE

 The first two for loops take Θ(n1 + n2 ) = Θ(n) time, where
n = r − p.
 The last for loop makes n iterations, each taking constant time, for
Θ(n) time.
 Total time: Θ(n).

Remark
 The test in line 14 is left-biased, which ensures that M ERGE -S ORT is
a stable sorting algorithm: if A[i] = A[j] and A[i] appears before A[j]
in the input array, then in the output array the element pointing to A[i]
appears to the left of the element pointing to A[j].
Characteristics of merge sort

 The worst-case running time of M ERGE -S ORT is Θ(n log n), much
better that the worst-case running time of I NSERTION -S ORT, which
was Θ(n2 ).
(see next slides for the explicit analysis of M ERGE -S ORT).
 M ERGE -S ORT is stable, because M ERGE is left-biased.
 M ERGE and therefore M ERGE -S ORT is not in-place:
it requires Θ(n) extra space.
 M ERGE -S ORT is not an online-algorithm: the whole array A must be
specified before the algorithm starts running.
Analysing divide-and-conquer algorithms [CLRS 2.3.2]

We often use a recurrence to express the running time of a

divide-and-conquer algorithm.
Let T (n) = running time on a problem of size n.
 If n is small (say n ≤ ℓ), use constant-time brute force solution.
 Otherwise, we divide the problem into a subproblems, each 1/b the
size of the original.
 Let the time to divide a size-n problem be D(n).
 Let the time to combine solutions (back to that of size n) be C(n).
We get the recurrence

c if n ≤ ℓ
T (n) =
a T (n/b) + D(n) + C(n) if n > ℓ
Example: M ERGE -S ORT

For simplicity, assume n = 2k .

For n = 1, the running time is a constant c.
For n ≥ 2, the time taken for each step is:
 Divide: Compute q = (p + r)/2; so, D(n) = Θ(1).
 Conquer: Recursively solve 2 subproblems, each of size n/2;
so, 2T (n/2).
 Combine: M ERGE two arrays of size n; so, C(n) = Θ(n).

More precisely, the recurrence for M ERGE -S ORT is


c if n = 1
T (n) =
2 T (n/2) + f (n) if n > 1

where the function f (n) is bounded as d′ n ≤ f (n) ≤ d n for suitable

constants d, d′ > 0.
Solving recurrence equations

We will consider three methods for solving recurrence equations:

1. Guess-and-test (called the substitution method in [CLRS])
2. Recursion tree
3. Master Theorem
4. By changing variables
Guess-and-test [CLRS 4.3]
 Guess an expression for the solution. The expression can contain
constants that will be determined later.
 Use induction to find the constants and show that the solution works.

Let us apply this method to M ERGE -S ORT.

The recurrence of M ERGE -S ORT implies that there exist two constants
c, d > 0 such that

c if n = 1
T (n) ≤
2 T (n/2) + d n if n > 1
Guess-and-test [CLRS 4.3]
 Guess an expression for the solution. The expression can contain
constants that will be determined later.
 Use induction to find the constants and show that the solution works.

Let us apply this method to M ERGE -S ORT.

The recurrence of M ERGE -S ORT implies that there exist two constants
c, d > 0 such that

c if n = 1
T (n) ≤
2 T (n/2) + d n if n > 1

Guess. There is some constant a > 0 such that T (n) ≤ an lg n for all
n ≥ 2 that are powers of 2.
Let’s test it!
Solving the M ERGE -S ORT recurrence by guess-and-test
Test. For n = 2k , by induction on k.
Base case: k = 1

T (2) = 2c + 2d ≤ a 2 lg 2 if a ≥ c + d

Inductive step: assume T (n) ≤ an log n for n = 2k .

Then, for n′ = 2k+1 we have:
n′ n′
′
T (n ) ≤ 2a 2 lg 2 + d n′

= an′ lg n′ − an′ lg 2 + d n′
≤ an′ lg n′ if a ≥ d

In summary: choosing a ≥ c + d ensures T (n) ≤ an lg n,

and thus T (n) = O(n log n).
A similar argument can be used to show that T (n) = Ω(n log n).
Hence, T (n) = Θ(n log n).
The recursion tree [CLRS 4.4]
Guess-and-test is great, but how do we guess the solution?
One way is to use the recursion tree,
which exposes successive unfoldings of the recurrence.

The idea is well exemplified in the case of M ERGE -S ORT.

The recurrence is

c if n = 1
T (n) =
2 T (n/2) + f (n) if n > 1

where the function f (n) satisfies the bounds d′ n ≤ f (n) ≤ d n, for

suitable constants d, d′ > 0.
Unfolding the recurrence of M ERGE -S ORT

Assume n = 2k for simplicity.

First unfolding: cost of f (n) plus cost of two subproblems of size n/2
f (n)

T (n/2) T (n/2)

Second unfolding: for each size-n/2 subproblem, cost of f (n/2) plus cost
of two subproblems of size n/4 each.
f (n)

f (n/2) f (n/2)

T (n/4) T (n/4) T (n/4) T (n/4)

Unfolding the recurrence of M ERGE -S ORT (cont’d)
Continue unfolding, until the problem size (= node label) gets down to 1:
f (n)

f (n/2) f (n/2)

f (n/4) f (n/4) f (n/4) f (n/4)

.. .. .. .. .. .. .. ..
. . . . . . . .
In total, there are lg n + 1 levels.
 Level 0 (root) has cost C0 (n) = f (n).
 Level 1 has cost C1 (n) = 2f (n/2).
 Level 2 has cost C2 (n) = 4f (n/4).
 For l < lg n, level l has cost Cl (n) = 2l f (n/2l ).
Note that, since d′ n ≤ f (n) ≤ d n, we have d′ n ≤ Cl (n) ≤ d n.
 The last level (consisting of n leaves) has cost cn.
Analysing M ERGE -S ORT with the recursion tree
The total cost of the algorithm is the sum of the costs of all levels:
lg n−1
X
T (n) = Cl (n) + c n .
l=0

Using the relation d′ n ≤ Cl (n) ≤ dn for l < lg n, we obtain the bounds

d′ n lg n + c n ≤ T (n) ≤ d n lg n + c n .

Hence, T (n) = Θ(n log n).

Recursion tree for
T(n) = 2T(n/2) + n2.
T(n) = T(n/3) + T(2n/3) + n.
The Master Theorem [DPV 2.2]

Theorem. Suppose

T (n) ≤ aT (⌈n/b⌉) + O(nd )

for some constants a > 0 and b > 1 and d ≥ 0.

Then, 
 O(nd ) if d > logb a
T (n) = O(nd logb n) if d = logb a
O(nlogb a ) if d < logb a


Example: For M ERGE -S ORT, a = b = 2 and d = 1.

The master theorem gives T (n) = O(n log n).
Note. See [CLRS 4.5] for a stronger version of the Master Theorem.
Proof of the Master Theorem⋆

By a recursion tree argument.

First assume n is a power of b. (We shall relax this later.)
The size of the subproblems decreases by a factor of b at each recursion,
and reaches the base case after logb n divisions.
Since the branching factor is a, level k of the tree comprises ak
subproblems, each of size n/bk .
Proof cont’d
The cost at level l is upper bounded by c al × ( bnl )d = c nd × ( bad )l ,
for a suitable constant c > 0.
Thus, the total cost is upper bounded by
 
d a  a  2  a  log b n
T (n) ≤ c n 1 + d + d + · · · + d .
b b b
 Proof cont’d
The cost at level l is upper bounded by c al × ( bnl )d = c nd × ( bad )l ,
for a suitable constant c > 0.
Thus, the total cost is upper bounded by
 
d a  a  2  a  log b n
T (n) ≤ c n 1 + d + d + · · · + d .
b b b
Now, there are three cases:
1. a < bd , i.e. d > logb a: the geometric series sums up to a constant.
Hence, T (n) = O(nd ).
2. a = bd , i.e. d = logb a: the geometric series sums up to 1 + logb n.
Hence, T (n) = O(nd log n). 

log n
3. a > bd , i.e. d < logb a: the geometric series sums up to Θ bad b
.
a logb n nlogb a


Since bd = log nad , we have
T (n) ≤ c nd Θ n ndb = Θ(nlogb a ). Hence, T (n) = O(nlogb a ).

DAA 2022 2. Divide and Conquer Algorithms – 22 / 60

 Further examples of divide-and-conquer algorithms

In the following, we will see divide-and-conquer algorithms for

 integer multiplication
 matrix multiplication
 search (in a sorted array)
 selection (finding the i-th smallest element in an array)

DAA 2022 2. Divide and Conquer Algorithms – 25 / 60