Decrease and Conquer

Decrease-and-Conquer
1. Reduce problem instance to smaller instance of the same problem
2. Solve smaller instance
3. Extend solution of smaller instance to obtain solution to original
instance

• Can be implemented either top-down or bottom-up

• Also referred to as inductive or incremental approach

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved.
Different Types
• Decrease by a constant (usually by 1):
• Insertion sort
• Topological sorting
• Algorithms for generating permutations, subsets

• Decrease by a constant factor (usually by half)

• Binary search and Bisection method
• Exponentiation by Squaring
• Multiplication à la russe

• Variable-size decrease
• Euclid’s algorithm
• Selection by partition
• Nim-like games
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved.
Computing an, Different strategies
• Brute Force:
• Multiply a, n times.

• Decrease by one:
• Compute an-1 and then multiply by a.

• Decrease by constant factor:

• Compute an/2 and then square them (For odd n>1, multiply it by a)

• Divide and conquer:

• Compute an/2 twice and then multiply them
Insertion Sort
To sort array A[0..n-1], sort A[0..n-2] recursively and then insert A[n-1]
in its proper place among the sorted A[0..n-2]

• Usually implemented bottom up (non-recursively)

Example: Sort 6, 4, 1, 8, 5

6|4 1 8 5
4 6|1 8 5
1 4 6|8 5
1 4 6 8|5
1 4 5 6 8
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ. 5
All Rights Reserved.
Pseudocode of Insertion Sort

Shift all elements in the sorted list

to create space for v

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 4 ©2012 Pearson Education, Inc. Upper Saddle River, NJ. A
ll Rights Reserved. 6
Analysis of Insertion Sort
• Time efficiency
Cworst(n) = n(n-1)/2  Θ(n2)
Cavg(n) ≈ n2/4  Θ(n2)
Cbest(n) = n - 1  Θ(n) (also fast on almost sorted arrays)

• Space efficiency: in-place

• Stability: yes

• Best elementary sorting algorithm overall

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.

• Binary insertion sort All Rights Reserved. 7

DAGs and Topological Sorting
A DAG: a directed acyclic graph, i.e. a directed graph with no
(directed) cycles
a b a b

a DAG not a DAG

c d c d

Arise in modeling many problems that involve prerequisite

constraints (construction projects, document version control)

Vertices of a DAG can be linearly ordered so that for every edge

its starting vertex is listed before its ending vertex (topological
sorting). Being a DAG is also a necessary condition for topological
sorting
A. Levitin betopossible.
“Introduction the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ. 8
All Rights Reserved.
Topological Sorting Example
Order the following items in a food chain
tiger

human

fish
sheep

shrimp

plankton wheat

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 9
Topological Sort Example
 Source Removal Algorithm
Source removal algorithm
Repeatedly identify and remove a source (a vertex with no
incoming edges) and all the edges incident to it until either no
vertex is left (problem is solved) or there is no source among
remaining vertices (not a DAG)
Example:
a b c d

e f g h

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.

Efficiency: same as efficiency of the DFS-based algorithm

All Rights Reserved. 13
Source-Removal Based Algorithm

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved.
 Decrease-by-Constant-Factor Algorithms

In this variation of decrease-and-conquer, instance size is

reduced by the same factor (typically, 2)

Examples:
• binary search and the method of bisection

• exponentiation by squaring

• multiplication à la russe (Russian peasant method)

• fake-coin puzzle
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 15
General Divide-and-Conquer
Recurrence
T(n) = aT(n/b) + f (n) where f(n)  (nd), d  0

Master Theorem: If a < bd, T(n)  (nd)

If a = bd, T(n)  (nd log n)
If a > bd, T(n)  (nlog b a )

Note: The same results hold with O instead of .

Examples: T(n) = 4T(n/2) + n  T(n)  ?

T(n) = 4T(n/2) + n2  T(n)  ?
T(n) = 4T(n/2) + n3  T(n)  ?

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 5 ©2012 Pearson Education, Inc. 16
Upper Saddle River, NJ. All Rights Reserved.
 Binary Search
Very efficient algorithm for searching in sorted
array:
K
vs
A[0] . . . A[m] . . . A[n-1]
If K = A[m], stop (successful search);
otherwise, continue
searching by the same method in A[0..m-1] if K
< A[m]
and in A[m+1..n-1] if K > A[m]

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 17
Binary Search -Analysis
• Basic Operation Using Master Theorem,
a=1, b=2, d=0
• Three way comparison Case of a = bd , So, the run time is :
• Recurrence Relation (Worst case) C(n)= O(nlog b a ), where logba = 0,
So, C(n) = O(log n)
• Analysis

•
Analysis of Binary Search
• This is VERY fast: e.g., Cw(106) = 20

• Optimal for searching a sorted array

• Limitations: must be a sorted array (not linked list)

• Bad (degenerate) example of divide-and-conquer

• Has a continuous counterpart called bisection method

for solving equations in one unknown f(x) = 0 (Taught in
Numerical Computing).

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 19
Exponentiation by Squaring
The problem: Compute an where n is a nonnegative integer

The problem can be solved by applying recursively the

formulas:

Base case
a 0= 1
For even values of n, (n >1)
a n = (a n/2 )2
For odd values of n, (n >1)
a n = (a (n-1)/2 )2.a
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 20
Exponentiation by Squaring-Analysis
• Basic Operation
• Multiplication
• Recurrence Relation
• = 1 or 2
• Analysis
• (for n= 2k for some k >0)

Using Master Theorem,

a=1, b=2, d=0
Case of a = bd , So, the run time is :
C(n)= O(nlog b a ), where logba = 0,
So, C(n) = O(log n)
Russian Peasant Multiplication
The problem: Compute the product of two positive integers

Can be solved by a decrease-by-half algorithm based on the following

For even values of n:
formulas.
*2m

For odd values of n:

*2m +m if n > 1

= 1 (Base case)

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 22
 Example of Russian Peasant
Multiplication
Compute 20 * 26
n m
20 26
10 52
5 104
2 208 + 104
1 416 416
520
Note: Method reduces to adding m’s values corresponding to odd n’s.

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 23
Russian Peasant Multiplication-
Analysis
• Basic Operation
• Shift operators + Addition (in some cases)
• Recurrence Relation

• Analysis
•
 Fake-Coin Puzzle (simpler version)
There are n identically looking coins one of which is fake.
There is a balance scale but there are no weights; the scale can tell whether
two sets of coins weigh the same and, if not, which of the two sets is heavier
(but not by how much). Design an efficient algorithm for detecting the fake
coin. Assume that the fake coin is known to be lighter than the genuine ones.

Decrease by factor 2 algorithm

Decrease by factor 3 algorithm

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 25
Divide into two Algorithm
• Divide the pile of n coins into two sets (set one coin aside if n is odd)
• Weigh them
• If equal, then the coin set aside is fake
• If not equal, solve the same problem with the lighter pile, until you reach the
end.

• Recurrence Relation Using Master Theorem,

Ww (n) = Ww() + 1 for n > 1, W(1) = 0 a=1, b=2, d=0
Case of a = bd , So, the run time is :
• Analysis C(n)= O(nlog b a ), where logba = 0,
So, C(n) = O(log n)
 Variable-Size-Decrease Algorithms
In the variable-size-decrease variation of decrease-and-
conquer, instance size reduction varies from one iteration to
another

Examples:
• Euclid’s algorithm for greatest common divisor
• partition-based algorithm for selection problem (Quickselect)
• interpolation search
• some algorithms on binary search trees
• Nim and Nim-like games
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 27
 Euclid’s Algorithm
Euclid’s algorithm is based on repeated application of
equality
gcd(m, n) = gcd(n, m mod n)

Ex.: gcd(80,44) = gcd(44,36) = gcd(36, 8) = gcd(8,4) =

gcd(4,0) = 4

One can prove that the size, measured by the second

number,
decreases at least by half after two consecutive Reason: n <iterations.
i+2 i n /2

Hence, T(n)  O(log n)

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 28
Selection Problem
Find the k-th smallest element in a list of n numbers
• k = 1 or k = n

• median: k = n/2
Example: 4, 1, 10, 9, 7, 12, 8, 2, 15 median = ?

The median is used in statistics as a measure of an

average value of a sample. In fact, it is a better (more
robust) indicator than the mean, which is used for the
same purpose.
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 30
Digression: Post Office Location Problem
Given n village locations along a straight highway, where
should a new post office be located to minimize the
average distance from the villages to the post office?

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 31
 Algorithms for the Selection Problem

The sorting-based algorithm: Sort and return the k-th element

Efficiency (if sorted by mergesort): Θ(nlog n)

A faster algorithm is based on the array partitioning:

s
all are ≤ A[s] all are ≥ A[s]

Assuming that the array is indexed from 0 to n-1 and s is a split

position obtained by the array partitioning:
If s = k-1, the problem is solved;
if s > k-1, look for the k-th smallest element in the left part;
if s < k-1, look for the (k-s)-th smallest element in the right part.
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
Note: The algorithm can simply continue until s = k-1.
All Rights Reserved. 32
 Two Partitioning Algorithms

There are two principal ways to partition an array:

• One-directional scan (Lomuto’s partitioning algorithm)

• Two-directional scan (Hoare’s partitioning algorithm)

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 33
 Lomuto’s Partitioning Algorithm

Scans the array left to right maintaining the array’s partition into
three contiguous sections: < p,  p, and unknown, where p is the
value of the first element (the partition’s pivot).
l s i r
p <p >= p ?

On each iteration the unknown section is decreased by one element

until it’s empty and a partition is achieved by exchanging the pivot
with the element in the split position s.
l s r
p <p >= p

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 34
 Tracing Lomuto’s Partioning Algorithm

s i
4 1 10 8 7 12 9 2 15
s i
4 1 10 8 7 12 9 2 15
s i
4 1 10 8 7 12 9 2 15
s i
4 1 2 8 7 12 9 10 15
s
4 1 2 8 7 12 9 10 15
2 1 4 8 7 12 9 10 15
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 4 ©2012 Pearson Education, Inc. Upper Saddle River, NJ. A
ll Rights Reserved. 35
Quickselect (based on Lomuto Partitioning)
 Tracing Quickselect (Partition-based
Algorithm)

Find the median of 4, 1, 10, 9, 7, 12, 8, 2, 15

Here: n = 9, k = 9/2 = 5, k -1=4
0 1 2 3 4 5 6 7 8
4 1 10 8 7 12 9 2 15

after 1st partitioning: s=2<k-1=4 2 1 4 8 7 12 9 10 15

after 2nd partitioning: s=4=k-1 8 7 12 9 10 15

7 8 12 9 10 15
The median is A[4]= 8

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 4 ©2012 Pearson Education, Inc. Upper Saddle River, NJ. A
ll Rights Reserved. 37
 Efficiency of Quickselect

Basic Operation
Comparison + Swaps
Average case (average split in the middle):
C(n) = C(n/2)+(n+1) C(n)  Θ(n)

Worst case (degenerate split): C(n)  Θ(n2)

A more sophisticated choice of the pivot leads to a

complicated algorithm with Θ(n) worst-case efficiency.

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 3 ©2012 Pearson Education, Inc. Upper Saddle River, NJ.
All Rights Reserved. 38
Summary
• Three variations of Decrease and Conquer
• Decrease by a constant
• Insertion Sort
• Topological Sort (Source Removal Algorithm)
• Decrease by a constant factor
• Binary Search
• Exponentiation by Squaring
• Fake coin example
• Russian Pheasant Multiplication
• Decrease by variable size
• Euclid Algorithm for GCD
• Quickselect