Divide-and-Conquer

Approach
Dr. Awnish Kumar
Assistant Professor
Computer Science and Engineering Department
National Institute of Technology Agartala
Recap of Insertion Sort
Insertion Sort – Best Case
• In INSERTION-SORT, the best case occurs when the array is already sorted.
• In this case, each time that line 5 executes, the value of key - the value originally in A[i] -
is already greater than or equal to all values in A[1:i-1], so that the while loop of lines 5 -
7 always exits upon the first test in line 5.
• Therefore, we have that ti = 1 for I = 2, 3, …, n, and the best-case running time is given by

• We can express this running time as (an + b) for constants a and b. The running time is
thus a linear function of n.
Insertion Sort – Worst Case
• The worst case arises when the array is in reverse sorted order - that is, it starts out in decreasing
order.
• The procedure must compare each element A[i] with each element in the entire sorted sub-array
A[1:i-1], and so ti = i for i = 2, 3, …, n. (The procedure finds that A[j] > key every time in line 5, and
the while loop exits only when j reaches 0.) Noting that

• We can express this worst-case running time as an2 + bn + c for constants a, b, and c. The running
time is thus a quadratic function of n.
Designing algorithms
• Insertion sort uses the incremental method: for each element A[i],
insert it into its proper place in the sub-array A[1:i], having already
sorted the sub-array A[1:i-1].
• We’ll use divide-and-conquer to design a sorting algorithm whose
worst-case running time is much less than that of insertion sort.
• One advantage of using an algorithm that follows the divide-and-
conquer method is that analyzing its running time is often
straightforward, using techniques that we’ll explore in coming slides.
The divide-and-conquer method
• Many useful algorithms are recursive in structure: to solve a given
problem, they recurse (call themselves) one or more times to handle
closely related sub-problems.
• These algorithms typically follow the divide-and-conquer method.
• In the divide-and-conquer method, if the problem is small enough - the
base case - you just solve it directly without recursing. Otherwise - the
recursive case -you perform three characteristic steps:
• Divide the problem into one or more sub-problems that are smaller
instances of the same problem.
• Conquer the sub-problems by solving them recursively.
• Combine the sub-problem solutions to form a solution to the original
problem.
Merge Sort
• Divide the subarray A[p:r] to be sorted into two adjacent subarrays,
each of half the size. To do so, compute the midpoint q of A[p:r]
(taking the average of p and r), and divide A[p:r] into subarrays A[p:q]
and A[q+1:r].
• Conquer by sorting each of the two subarrays A[p:q] and A[q+1:r]
recursively using merge sort.
• Combine by merging the two sorted subarrays A[p:q] and A[q+1:r]
back into A[p:r], producing the sorted answer.
Merge Sort
Analyzing divide-and-conquer algorithms
• When an algorithm contains a recursive call, you can often describe its running time by a
recurrence equation or recurrence, which describes the overall running time on a problem of size
n in terms of the running time of the same algorithm on smaller inputs.
• let T(n) be the worst-case running time on a problem of size n.
• If the problem size is small enough, say n < n0 for some constant n0 > 0, the straightforward
solution takes constant time, which we write as θ(1).
• Suppose that the division of the problem yields a sub-problems, each with size n=b, that is, 1/b
the size of the original.
• For merge sort, both a and b are 2, but we’ll see other divide-and-conquer algorithms in which a ≠
b.
• It takes T(n/b) time to solve one sub-problem of size n=b, and so it takes aT(n/b) time to solve all
a of them.
• If it takes D(n) time to divide the problem into sub-problems and C(n) time to combine the
solutions to the sub-problems into the solution to the original problem, we get the recurrence
Analysis of merge sort
• Sometimes, the n/b size of the divide step isn’t an integer.
• For example, the MERGE-SORT procedure divides a problem of size n
into sub-problems of sizes floor(n/2) and ceil(n/2). Since the
difference between floor(n/2) and ceil(n/2) is at most 1, we’ll squint a
little and just call them both size n=2.
• This simplification of ignoring floors and ceilings does not generally
affect the order of growth of a solution to a divide-and-conquer
recurrence.
Analysis of merge sort
• Divide: The divide step just computes the middle of the subarray, which
takes constant time. Thus, D(n) = θ(1).
• Conquer: Recursively solving two subproblems, each of size n=2,
contributes 2T(n\2) to the running time (ignoring the floors and ceilings, as
we discussed).
• Combine: Since the MERGE procedure on an n-element subarray takes θ(n)
time, we have C(n) = θ(n).
• The recurrence for the worst-case running time T(n) of merge sort:
T(n) = 2T(n/2) + θ (n).
• Using Master Theorem, T(n) = θ(n lg n).
• For large enough inputs, merge sort, with its θ (n lg n) worst-case running
time, outperforms insertion sort, whose worst-case running time is θ (n2)
Recursion Tree
Solving Recurrences
• Iteration Method
• Substitution Method
• Recursion-tree Method
• Master Method
Recurrences and Running Time
• An equation or inequality that describes a function in terms of
its value on smaller inputs.
T(n) = T(n-1) + n
• Recurrences arise when an algorithm contains recursive calls to
itself

• What is the actual running time of the algorithm?

• Need to solve the recurrence

• Find an explicit formula of the expression
• Bound the recurrence by an expression that involves n

14
Example Recurrences
• T(n) = T(n-1) + n Θ(n2)
• Recursive algorithm that loops through the input to eliminate one item
• T(n) = T(n/2) + c Θ(lgn)
• Recursive algorithm that halves the input in one step
• T(n) = T(n/2) + n Θ(n)
• Recursive algorithm that halves the input but must examine every item in the
input
• T(n) = 2T(n/2) + 1 Θ(n)
• Recursive algorithm that splits the input into 2 halves and does a constant
amount of other work

15
Recurrent Algorithms
BINARY-SEARCH
• for an ordered array A, finds if x is in the array A[lo…hi]

Alg.: BINARY-SEARCH (A, lo, hi, x)

1 2 3 4 5 6 7 8

if (lo > hi) 2 3 5 7 9 10 11 12

return FALSE
mid  (lo+hi)/2 lo
mid
hi
if x = A[mid]
return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x)
if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x)
16
Example
• A[8] = {1, 2, 3, 4, 5, 7, 9, 11}
• lo = 1 hi = 8 x = 7
1 2 3 4 5 6 7 8

1 2 3 4 5 7 9 11 mid = 4, lo = 5, hi = 8

5 6 7 8

1 2 3 4 5 7 9 11 mid = 6, A[mid] = x Found!

17
Another Example
• A[8] = {1, 2, 3, 4, 5, 7, 9, 11}
– lo = 1 hi = 8 x=6
1 2 3 4 5 6 7 8

1 2 3 4 5 7 9 11 mid = 4, lo = 5, hi = 8
low high

1 2 3 4 5 7 9 11 mid = 6, A[6] = 7, lo = 5, hi = 5
low high

1 2 3 4 5 7 9 11 mid = 5, A[5] = 5, lo = 6, hi = 5
NOT FOUND!

1 2 3 4 5 7 9 11
high low

18
Analysis of BINARY-SEARCH
Alg.: BINARY-SEARCH (A, lo, hi, x)
if (lo > hi) constant time: c1
return FALSE
mid  (lo+hi)/2 constant time: c2

if x = A[mid] constant time: c3

return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x) same problem of size n/2

if ( x > A[mid] )
same problem of size n/2
BINARY-SEARCH (A, mid+1, hi, x)

• T(n) = c + T(n/2)
• T(n) – running time for an array of size n
19
Methods for Solving Recurrences

• Iteration method

• Substitution method

• Recursion tree method

• Master method

20
The Iteration Method
• Convert the recurrence into a summation and try to bound it using
known series
• Iterate the recurrence until the initial condition is reached.
• Use back-substitution to express the recurrence in terms of n and the initial
(boundary) condition.

21
The Iteration Method
T(n) = c + T(n/2) T(n/2) = c + T(n/4)
T(n) = c + T(n/2) T(n/4) = c + T(n/8)
= c + c + T(n/4)
= c + c + c + T(n/8)
Assume n = 2k
T(n) = c + c + … + c + T(1)
k times
= clgn + T(1)
= Θ(lgn)
22
Iteration Method – Example
T(n) = n + 2T(n/2) Assume: n = 2k
T(n) = n + 2T(n/2) T(n/2) = n/2 + 2T(n/4)
= n + 2(n/2 + 2T(n/4))
= n + n + 4T(n/4)
= n + n + 4(n/4 + 2T(n/8))
= n + n + n + 8T(n/8)
… = in + 2iT(n/2i)
= kn + 2kT(1)
= nlgn + nT(1) = Θ(nlgn)

23
The substitution method

1. Guess a solution

2. Use induction to prove that the

solution works

24
Substitution method
• Guess a solution
• T(n) = O(g(n))
• Induction goal: apply the definition of the asymptotic notation
• T(n) ≤ d g(n), for some d > 0 and n ≥ n0 (strong induction)

• Induction hypothesis: T(k) ≤ d g(k) for all k < n

• Prove the induction goal

• Use the induction hypothesis to find some values of the constants d and n0 for which the
induction goal holds

25
Example: Binary Search
T(n) = c + T(n/2)
• Guess: T(n) = O(lgn)
• Induction goal: T(n) ≤ d lgn, for some d and n ≥ n0
• Induction hypothesis: T(n/2) ≤ d lg(n/2)

• Proof of induction goal:

T(n) = T(n/2) + c ≤ d lg(n/2) + c
= d lgn – d + c ≤ d lgn
if: – d + c ≤ 0, d ≥ c
• Base case?
26
Example 2
T(n) = T(n-1) + n
• Guess: T(n) = O(n2)
• Induction goal: T(n) ≤ c n2, for some c and n ≥ n0
• Induction hypothesis: T(n-1) ≤ c(n-1)2 for all k < n

• Proof of induction goal:

T(n) = T(n-1) + n ≤ c (n-1)2 + n
= cn2 – (2cn – c - n) ≤ cn2
if: 2cn – c – n ≥ 0  c ≥ n/(2n-1)  c ≥ 1/(2 – 1/n)
• For n ≥ 1  2 – 1/n ≥ 1  any c ≥ 1 will work

27
Example 3
T(n) = 2T(n/2) + n
• Guess: T(n) = O(nlgn)
• Induction goal: T(n) ≤ cn lgn, for some c and n ≥ n0
• Induction hypothesis: T(n/2) ≤ cn/2 lg(n/2)

• Proof of induction goal:

T(n) = 2T(n/2) + n ≤ 2c (n/2)lg(n/2) + n
= cn lgn – cn + n ≤ cn lgn
if: - cn + n ≤ 0  c ≥ 1
• Base case? 28
Changing variables
T(n) = 2T( n ) + lgn
• Rename: m = lgn  n = 2m
T (2m) = 2T(2m/2) + m
• Rename: S(m) = T(2m)
S(m) = 2S(m/2) + m  S(m) = O(mlgm)
(demonstrated before)
T(n) = T(2m) = S(m) = O(mlgm)=O(lgnlglgn)
Idea: transform the recurrence to one that you have
seen before

29
The recursion-tree method

Convert the recurrence into a tree:

• Each node represents the cost incurred at various levels
of recursion

• Sum up the costs of all levels

Used to “guess” a solution for the recurrence

30
Example 1
W(n) = 2W(n/2) + n 2

• Subproblem size at level i is: n/2i

• Subproblem size hits 1 when 1 = n/2i  i = lgn
• Cost of the problem at level i = (n/2i)2 No. of nodes at level i = 2i
• Total cost: lg n 1 2
n lg n 1 i
1 
1
i
1
W (n)   i  2 W (1)  n     n  n     O(n) n 2
lg n 2 2
 O ( n)  2n 2
i 0 2 i 0  2  i 0  2  1 1
2
 W(n) = O(n2) 31
Example 2
E.g.: T(n) = 3T(n/4) + cn2

• Subproblem size at level i is: n/4i

• Subproblem size hits 1 when 1 = n/4i  i = log4n
• Cost of a node at level i = c(n/4i)2
• Number of nodes at level i = 3i  last level has 3log4n = nlog43 nodes
• Total cost:
log4 n 1 i

     
 i
3 2 3 1
T ( n)     cn   n
 16 
log4 3
    cn 2   n log4 3 
i  0  16 
3
cn 2   n log4 3  O(n 2 )
i 0
1
16
 T(n) = O(n ) 2 32
Example 2 - Substitution
T(n) = 3T(n/4) + cn 2

• Guess: T(n) = O(n2)

• Induction goal: T(n) ≤ dn2, for some d and n ≥ n0
• Induction hypothesis: T(n/4) ≤ d (n/4)2

• Proof of induction goal:

T(n) = 3T(n/4) + cn2
≤ 3d (n/4)2 + cn2
= (3/16) d n2 + cn2
≤ d n2 if: d ≥ (16/13)c

• Therefore: T(n) = O(n2)

33
Example 3 (simpler proof)
W(n) = W(n/3) + W(2n/3) + n

• The longest path from the root to

a leaf is:
n  (2/3)n  (2/3)2 n  …  1

• Subproblem size hits 1 when

1 = (2/3)in  i=log3/2n

• Cost of the problem at level i = n

• Total cost:
lg n
W (n)  n  n  ...  n(log3/ 2 n)  n  O(n lg n)
3
lg
2
 W(n) = O(nlgn) 34
Example 3
W(n) = W(n/3) + W(2n/3) + n

• The longest path from the root to

a leaf is:
n  (2/3)n  (2/3)2 n  …  1

• Subproblem size hits 1 when

1 = (2/3)in  i=log3/2n

• Cost of the problem at level i = n

• Total cost:
(log3 / 2 n ) 1
W (n)  n  n  ...  i 0
n  2(log3 / 2 n )W (1) 
log3 / 2 n
lg n 1
n 
i 0
1  nlog3 / 2 2  n log3/ 2 n  O(n)  n
lg 3/ 2
 O ( n) 
lg 3/ 2
n lg n  O(n)

 W(n) = O(nlgn) 35
Example 3 - Substitution
W(n) = W(n/3) + W(2n/3) + O(n)
• Guess: W(n) = O(nlgn)
• Induction goal: W(n) ≤ dnlgn, for some d and n ≥ n0
• Induction hypothesis: W(k) ≤ d klgk for any K < n
(n/3, 2n/3)
• Proof of induction goal:
Try it out as an exercise!!
• T(n) = O(nlgn)

36
Master Theorem
• If the recurrence is of the form ,
where a ≥ 1,b > 1,k ≥ 0 and p is a real number, then:
Master Theorem
Master Theorem
Master Theorem
Master Theorem for Subtract and Conquer
Recurrences
Multiplying square matrices

• Because each of the triply nested for loops runs for exactly n iterations, and
each execution of line 4 takes constant time, the MATRIX-MULTIPLY
procedure operates in θ(n3) time.
• Even if we add in the θ(n2) time for initializing C to 0, the running time is
still θ(n3).
Multiplying square matrices: D&C Approach
• For n > 1, the divide step partitions the (n x n) matrices into four (n/2
x n/2) sub-matrices.
• We’ll assume that n is an exact power of 2, so that as the algorithm
recurses, we are guaranteed that the submatrix dimensions are
integer.
Multiplying square matrices: D&C Approach
Multiplying square matrices: D&C Approach
• There are two common approaches for implementing the matrix
partitioning.
• One strategy is to allocate temporary storage to hold A’s four sub-matrices
A11, A12, A21, and A22 and B’s four sub-matrices B11, B12, B21, and B22.
• Then copy each element in A and B to its corresponding location in the
appropriate sub-matrix.
• After the recursive conquer step, copy the elements in each of C’s four sub-
matrices C11, C12, C21, and C22 to their corresponding locations in C .
• This approach takes θ(n2) time.
Multiplying square matrices: D&C Approach
• The second approach uses index calculations and is faster and more
practical.
• A sub-matrix can be specified within a matrix by indicating where
within the matrix the sub-matrix lies without touching any matrix
elements.
• Partitioning a matrix (or recursively, a sub-matrix) only involves
arithmetic on this location information, which has constant size
independent of the size of the matrix.
• This approach takes θ(1) time.
Multiplying square matrices: D&C Approach
Multiplying square matrices: D&C Approach
• Recurrence for the running time of MATRIX-MULTIPLY-RECURSIVE is
T (n) = 8T(n/2) + θ(1)
• From the master method, the above recurrence (4.9) has the solution
T (n) = θ(n3), which means that it has the same asymptotic running
time as the straightforward MATRIX-MULTIPLY procedure.
Strassen’s algorithm for matrix multiplication
• Many mathematicians presumed that it was not possible to multiply
matrices in O(n3) time until 1969, when V. Strassen published a remarkable
recursive algorithm for multiplying (n x n) matrices.
• Strassen’s algorithm runs in θ(nlg 7) time. Since lg 7 = 2.8073549…,
Strassen’s algorithm runs in O(n2.81) time, which is asymptotically better
than the θ(n3) MATRIX-MULTIPLY and MATRIX-MULTIPLY-RECURSIVE
procedures.
• Instead of performing eight recursive multiplications of (n/2 x n/2)
matrices, Strassen’s algorithm performs only seven.
• The cost of eliminating one matrix multiplication is several new additions
and subtractions of (n/2 x n/2) matrices, but still only a constant number.
Strassen’s algorithm for matrix multiplication
• To get an idea how the number of multiplications might be reduced, as well as why
reducing the number of multiplications might be desirable for matrix calculations,
suppose that you have two numbers x and y, and you want to calculate the quantity x2 -
y 2.
• The straightforward calculation requires two multiplications to square x and y, followed
by one subtraction (which you can think of as a negative addition).
• But let’s recall the old algebra trick x2 - y2 = x2 - xy + xy - y2 = x(x – y) + y(x – y) = (x + y)(x –
y).
• Using this formulation of the desired quantity, you could instead compute the sum (x + y)
and the difference (x – y) and then multiply them, requiring only a single multiplication
and two additions.
• At the cost of an extra addition, only one multiplication is needed to compute an
expression that looks as if it requires two. If x and y are scalars, there’s not much
difference: both approaches require three scalar operations.
• If x and y are large matrices, however, the cost of multiplying outweighs the cost of
adding, in which case the second method outperforms the first method.
Strassen’s algorithm for matrix multiplication
Strassen’s algorithm for matrix multiplication
Strassen’s algorithm for matrix multiplication
Strassen’s algorithm for matrix multiplication
Problems
• Problem 1: Let us consider an algorithm A which solves problems by
dividing them into five subproblems of half the size, recursively solving
each subproblem, and then combining the solutions in linear time. What is
the complexity of this algorithm?
• Solution: T(n) = 5T(n/2) + O(n) =>

• Problem 2: An algorithm B solves problems of size n by recursively solving

two subproblems of size n – 1 and then combining the solutions in constant
time. What is the complexity of this algorithm?
• Solution: T(n) = 2T(n-1) + O(1) =>
Problems and Solutions
• Problem 3: An algorithm C solves problems of size n by dividing them
into nine subproblems of size n/3 , recursively solving each
subproblem, and then combining the solutions in O(n2) time. What is
the complexity of this algorithm?
• Solution: T(n) = 9T(n/3) + O(n2) => T(n) = O(n2log n)
• Problem 4: Write a recurrence and solve it.
• Solution: T(n) = 2T(n/2) + O(1)
=> T(n) =
Problems and Solutions
• Problem 5: Discuss binary search and its time complexity.
• Solution: T(n) = T(n/2) + O(1)
=> T(n) = O(log n)
• Problem 6: Consider the modified version of binary search. Let us assume
that the array is divided into 3 equal parts (ternary search) instead of 2
equal parts. Write the recurrence for this ternary search and find its
complexity.
• Solution: T(n) = T(n/3) + O(1)
• Problem 7: what if we divide the array into two sets of sizes approximately
one-third and two-thirds.
• Solution: T(n) = T(2n/3) + O(1)
Problems and Solutions
• Problem 8: We are given two sorted lists of size n. Give an algorithm for finding
the median element in the union of the two lists.
• Solution: We use the Merge Sort process. Use merge procedure of merge sort.
Keep track of the count while comparing elements of two arrays. If the count
becomes n (since there are 2n elements), we have reached the median. Take the
average of the elements at indexes n – 1 and n in the merged array. T(n) = O(n).
• Problem 9: Can we give the algorithm if the size of the two lists are not the same?
• Solution: The solution is similar to the previous problem. Let us assume that the
lengths of two lists are m and n. In this case we need to stop when the counter
reaches (m + n)/2.
Time Complexity: T(n) = O((m+n)/2).
Problems and Solutions
• Problem 10: Can we improve the time-complexity of previous problem to O(log n)?
• Solution: Yes, using the D & C approach. Let us assume that the given two lists are L1 and L2.
• Algorithm:
1. Find the medians of the given sorted input arrays L1[] and L2[]. Assume that those medians are m1 and m2.
2. If m1 and m2 are equal then return m1 (or m2).
3. If m1 is greater than m2, then the final median will be below two sub arrays.
4. From first element of L1 to m1.
5. From m2 to last element of L2.
6. If m2 is greater than m1, then median is present in one of the two sub arrays below.
7. From m1 to last element of L1.
8. From first element of L2 to m2.
9. Repeat the above process until the size of both the sub arrays becomes 2.
10. If size of the two arrays is 2, then use the formula below to get the median.
11. Median = (max(L1[0],L2[0]) + min(L1[1],L2[1])/2
Time Complexity: O(logn) since we are considering only half of the input and throwing the remaining half.
Maximum Value Contiguous Subsequence
• Problem 11: Given a sequence of n numbers A(1) ...A(n), give an
algorithm for finding a contiguous subsequence A(i) ...A(j) for which
the sum of elements in the subsequence is maximum. Example : {-2,
11, -4, 13, -5, 2} → 20 and {1, -3, 4, -2, -1, 6 } → 7.
• Solution: Divide this input into two halves. The maximum contiguous
subsequence sum can occur in one of 3 ways:
• Case 1: It can be completely in the first half
• Case 2: It can be completely in the second half
• Case 3: It begins in the first half and ends in the second half
Maximum Value Contiguous Subsequence
• Recursively compute the maximum contiguous
subsequence that resides entirely in the first half.
• Recursively compute the maximum contiguous
subsequence that resides entirely in the second
half.
• Compute, via two consecutive loops, the
maximum contiguous subsequence sum that
begins in the first half but ends in the second
half.
• Choose the largest of the three sums.
• The program performs two recursive calls plus
the linear work involved in computing the
maximum sum for case 3. The recurrence
relation is:

• Using D & C Master theorem, we get the time

complexity as T(n) = O(nlogn).
Long Integer Multiplication
• https://codecrucks.com/large-integer-multiplication-using-divide-
and-conquer/
End of Module 2