Unit 1.

Sorting and
Divide and Conquer
Lecture 1 Introduction to
Algorithm and Sorting
 What is an Algorithm?
• An algorithm is a computational procedure
that takes some value, or a set of values,
as input and produces some values, or a
set of values, as output.
• So, it is a sequence of computational
steps that transform the input into the
output.
• Correction: sequence -> combination
 Sequential and Parallel
• In sequential computation, an algorithm is
a sequence of computational steps.

• However, it is not true in parallel

computation.

• In this course, we study only algorithms in

sequential computation.
 Algorithms for Sorting
Input : A sequence of n numbers {a1 , a2 ,..., an }.
Output : A permutation {a '1 , a '2 ,..., a 'n } of input
sequence such that a '1 a '2  a 'n .
e.g.,
Input : 5, 2, 4, 6, 1, 3.
Output :1, 2, 3, 4, 5, 6.
Insertion Sort, Merge Sort
 Efficiency
• Running time from receiving the input to
producing the output.

Running time
2
Insert Sort O(n )
Merge Sort O(n log n)
 f (n) O( g (n)) means
there exist constants c  0 and n0  0
such that
0  f (n) cg (n) for n n0 .
c1n 2 and c2 n log n, who is bigger?

When n is sufficiently large,

c1n 2  c2 n log n.

Therefore, what we really care about is the growth of the

function!
 Insertion Sort
input array A
for j  2 to length[ A]
do key  A[ j ]
i j 1
while i  0 and A[i ]  key
do A[i  1]  A[i ]
i i 1
A[i  1]  key
for j  2 to length[ A]
do begin
key  A[ j ];
i  j  1;
while i  0 and A[i ]  key
do begin
A[i  1]  A[i ];
i  i  1;
end - while;
A[i  1]  key;
end - for.
 key
  
5, 2, 4, 6, 1, 3 2, 4, 5, 6, 1 , 3 1, 2, 4, 5, 6, 3
  
2, 5, 4, 6, 1, 3 2, 4, 5, 1 , 6, 3 1, 2, 4, 5, 3, 6
 
2, 4, 1 , 5, 6, 3 1, 2, 4, 3, 5, 6
 
2, 5, 4, 6, 1, 3 2, 1 , 4, 5, 6, 3 
  1, 2, 3, 4, 5, 6
2, 4, 5, 6, 1, 3 1 , 2, 4, 5, 6, 3


2, 4, 5, 6, 1, 3 In the key outside of array.
 How to calculate running time?
• Each “line” of pseudocode requires a
constant time. (In RAM model)
for j  2 to length[ A] This loop runs n-1 times
do begin and each time runs at most
4+3(j-1) lines.
key  A[ j ];
i  j  1;
while i  0 and A[i ]  key
do begin This loop runs at most
j-1 times and each time
A[i  1]  A[i ]; runs at most 3 lines.
i  i  1;
end - while; T (n)  j 2 (4  3( j  1))
n

A[i  1]  key; (n  1)(n  2)

n  1  3
end - for. 4(n-1) 2
 Remark on Running Time
• Running time is a function of input size.
• In Turing machine model, the input size
of sorting is

log 2 a1  log 2 a2    log 2 an

Hence, " A[i ]  key" runs not in constant time.
 Divide and Conquer
• Divide the problem into subproblems.
• Conquer the subproblems by solving
them recursively.
• Combine the solutions to subproblems
into the solution for original problem.
 Merge Sort
Let Merge - Sort ( A, p, r ) be a procedure
which sorts elements in subarray A[ p...r ].

Main Program
begin
Merge - Sort ( A,1, n);
end
 Procedure
Merge - Sort ( A, p, r )
if p  r
then begin q  ( p  r ) / 2;
Merge - Sort ( A, p, q );
Merge - Sort ( A, q  1, r );
Merge( A, p, q, r );
end - then.
 5,2,7,4,6,8,1,3

5,2,7,4 6,8,1,3

5,2 7,4 6,8 1,3

5 2 7 4 6 8 1 3

2,5 4,7 6,8 1,3

2,4,5,7 1,3,6,8

1,2,3,4,5,6,7,8
Merge( A, p, q, r )
n1  q  p  1;
n2  r  q;
create array L[1..n1  1] and R[1..n2  1];
for i  1 to n1
do L[i ]  A[ p  i  1];
for j  1 to n2
do R[ j ]  A[q  j ];
L(n1  1)  ; R (n2  1)  ;
i  1;
j  1;
for k  p to r
do if L[i ] R[ j ]
then A[k ]  L[i ] and i  i  1
else A[k ]  R[ j ] and j  j  1;
  
Example L : 2, 3, 5, 7, ; R : 1 , 4, 6, 8, ;
A : 1,
 
L : 2, 3, 5, 7, ; R :1, 4, 6, 8, ;
A :1, 2,
 
L : 2, 3, 5, 7, ; R :1, 4, 6, 8, ;
A :1, 2, 3,
 
L : 2, 3, 5, 7, ; R :1, 4, 6, 8, ;
A :1, 2, 3, 4,
 
L : 2, 3, 5, 7, ; R :1, 4, 6, 8, ;
A :1, 2, 3, 4, 5,
  
L : 2, 3, 5, 7, ; R :1, 4, 6, 8, ;
A : 1, 2, 3, 4, 5, 6,
 
L : 2, 3, 5, 7, ; R :1, 4, 6, 8, ;
A :1, 2, 3, 4, 5, 6, 7,
 
L : 2, 3, 5, 7, ; R :1, 4, 6, 8, ;
A :1, 2, 3, 4, 5, 6, 7, 8,
 
L : 2, 3, 5, 7, ; R :1, 4, 6, 8, ;
 Procedure
Merge - Sort ( A, p, r ) T (n)
if p  r
then begin q  ( p  r ) / 2;
T (  n / 2 )
Merge - Sort ( A, p, q );
Merge - Sort ( A, q  1, r ); T (  n / 2 )
Merge( A, p, q, r ); (n)
end - then.
T (n) 2T (  n / 2 )  (n)
 f (n) ( g (n)) means
there exist constants 0 c1 c2 and n0  0
such that
0 c1 g (n)  f (n) c2 g (n) for n n0 .
Symmetry

f (n) ( g (n))  g (n) ( f (n))

 How to Solve Recurrences
• Recursion-tree method. (give reasonable
estimate)
• Substitution method. (mathematical
induction and find correct constant)
• Master method. (already proved results)
 Recursion Tree
T (2 k ) c2 2 k

T (2 k  1 ) T (2 k  1 ) 2c2 2 k  1 c2 2 k

T (2 k  2 ) T (2 k  2 ) T (2 k  2 ) T (2 k  2 )

T (1) T (1) T (1) T (1)

T (2 k ) c2 k 2 k
 Substitution Method
• Guess the form of the solution (e.g., use
recursive tree).
• Use math induction to find the constants
and also show the solution works.
Solve T (n) 2T (  n / 2 )  (n)

Note : 2T (n / 2)  c1n T (n) 2T (n / 2)  c2 n

for some constants c2 c1  0, n n0 .
Guess : T (n) cn lg n, i.e., T (n) O(n lg n)
Induction Proof (analysis) :
(1) For n n0 , T (n) T (n0 ).
(2) Assume T (n / 2) c(n / 2) lg(n / 2).
Then T (n) 2c(n / 2) lg(n / 2)  c2 n
cn lg n  (c2  c)n.
cn lg n
(Take c max(c2 , T (n0 )).)
 Conclusion

If T (n) 2T (n / 2)  O(n)

then T (n) O(n lg n).
 Master Theorem
Let T (n) aT (n / b)  f (n) where constants
a 1, b  1, and n / b means  n / b  or  n / b .
If f (n) O(n logb a   ) for some constant   0,
then T (n) (n logb a ).
If f (n) (n logb a ), then T (n) (n logb a lg n).
log b a 
If f (n) (n ) for some constant   0,
and if af (n / b) cf (n) for a constant c  1 and
sufficient ly large n, then T (n) ( f (n)).
 f (n) ( g (n)) means
there exist constants c  0 and n0  0
such that
0 cg (n)  f (n) for n n0 .
Relationship

f (n) ( g (n))  g (n) O( f (n))

 Summary
f (n) O( g (n)) means
(c2  0)(n0  0)(n n0 ) f (n) c2 g (n).
f (n) ( g (n)) means
(c1  0, c2  0)(n0  0)(n n0 ) c1 g (n)  f (n) c2 g (n).
f (n) ( g (n)) means
(c1  0)(n0  0)(n n0 ) c1 g (n)  f (n).

f (n) ( g (n))  g (n) O( f (n))

 What we learnt in this lecture?
• How to calculate running time.
• How to solve recurrences.
• Insertion sort and Merge sort.
• Divide and conquer
• Lecture Notes give you key points in
each lecture.
• You must read textbook after lectures
in order to study well.
Solve T (n) 2T (  n / 2 )  (n)

T (n) 2T (  n / 2 )  (n) means

(c1  0, c2  0)(n0  0)(n n0 )
2T (  n / 2 )  c1n T (n) 2T (  n / 2 )  c2 n
Choose c2 max(T (1), T (2),...,T (n0 )).
Then we have
(n 1) T (n) 2T (  n / 2 )  c2 n.

We start to work from here.

We will prove T (n) 10c2 n lg n by mathematical induction :
(1) For n 2, T (1) 0. T (2) 2c2 . Where it comes from?

(2) Assume T (  n / 2 ) 10c2  n / 2 lg  n / 2.

Then for n 3, T (n) 20c2  n / 2 lg  n / 2  c2 n
20c2 (n / 2  1 / 2)(lg n  1  lg e / n)  c2 n
10c2 (n  1)(lg n  0.5)  c2 n (lg e / n  0.5 for n 3)
10c2 n lg n

lg  n / 2 lg(n / 2  1 / 2)
lg(n / 2)  lg(1  1 / n)
lg n  1  lg e1/ n (note : 1  x e x )
lg n  1  (lg e) / n
 Where 10c2 comes from?

Guess : T (n) cn lg n, i.e., T (n) O(n lg n)

Prove by Induction (analysis):
(1) For n 2, T (1) 0. T (2) 2c2 2c. (Require c c2 .)
(2) Assume T (  n / 2 ) c  n / 2 lg  n / 2.
Then for n 3, T (n) 2c  n / 2 lg  n / 2  c2 n
2c(n / 2  1 / 2)(lg n  1  lg e / n)  c2 n
c(n  1)(lg n  0.5)  c2 n (lg e / n  0.5 for n 3)
cn lg n  (c2  (0.5  (lg n  0.5) / n)c)n
cn lg n  (c2  0.1c)n
cn lg n (require c 10c2 )
(Take c 10c2 .)