Analysis of Algorithms

Types of Sorting Algorithms

● Non-recursive comparison sorting
■ Insertion sort
■ Bubble sort
■ Selection sort
● Recursive comparison sorting
■ Merge sort
■ Quick sort
■ Heap sort
● Non-comparison linear sorting
■ Count sort
■ Radix sort
2
■ Bucket sort
 Recursive Comparison Sorting
● There are many ways to design algorithms
● Insertion/bubble/selection sort uses an incremental approach
■ Having sorted to array A[i..j]
■ We insert the single element A[j] into its proper
place, to get a sorted array A[i..j]

● An alternative design approach

■ “Divide and Conquer”

3
 Divide and Conquer Approach
● Divide the problems into a number of sub problems.

● Conquer the sub problems by solving them recursively. If

the sub-problem sizes are small enough, just solve the
problems in a straight forward manner.

● Combine the solutions to the sub problems into the solution

for the original problem.

4
 Merge sort
● Merge sort: orders a list of values by recursively dividing the list in half
until each sub-list has one element, then recombining
■ Invented by John von Neumann in 1945
● More specifically:
■ Divide the n element sequence to be sorted into two subsequences of
n/2 elements each.
■ Conquer: Sort the two subsequences to produce the sorted answer.
■ Combine: Merge the two sorted sub sequences to produce the sorted
answer.

mergeSort(0, mergeSort(n/2,
n/2-1) n-1)
5
sor merge(0, n/2, sor
t n-1) t
 Merge sort
Base Case: When the sequences to be sorted has length 1.

Unsorted
Sorted 66
12 108
14 56
34 14
56 89
66 12
89 108
34

Divide
Merge Divide
Merge
66
14 108
56 56
66 108
14 89
12 12
34 34
89

Divide
Merge Divide
Merge Divide
Merge Divide
BCase
Merge
66 108 56
14 14
56 89
12 12
89 34
66 108

Divide
BCase
Merge Divide
BCase
Merge Divide
BCase
Merge Divide
BCase
Merge
6
56 89 12
66 108 14
Merge Sort Pseudo Code

7
Merge Sort Pseudo Code (2)

8
 Merging Algorithm
● The merge operation:
■ Given two sorted arrays, merge operation produces a
sorted array with all the elements of the two arrays
■ Temporary array is used to store the sorted elements
from the two arrays

A 6 13 18 21 B 4 8 9 20

C 4 6 8 9 13 18 20 21

● Running time of merge: O(n), where n is the number of

9
elements in the merged array.
 Mergesort Analysis
● Let T(n) be the time taken by this algorithm to sort an array
of n elements dividing A into sub-arrays A1 and A2.

● It is easy to see that the Merge (A1, A2, A) takes the linear
time. Consequently,

T(n) = T(n/2) + T(n/2) + θ(n)

T(n) = 2T (n/2) + θ(n)

● The above recurrence relation is non-homogenous and can

be solved by any of the methods:
■ Substitution
■ Recursion tree 10
■ Master method
 Mergesort Analysis
Level 0 Merge n items:
n O(n)

n/ n/ Level 1 Merge two n/2

2 2 items:
O(n)
n/ n/ n/ n/ Level 2
4 4 4 4
Each level requires
O(n) operations

1 1 1 ... 1 1 1 Tree Height : log2n

11
Each level O(n) operations & O(log2n) levels 🡪
O(n*log2n)
 Mergesort Analysis

● Worst case: O(n * log2n).

● Average case: O(n * log2n).

● Performance is independent of the initial order of

the array items.

● Advantage:
■ Mergesort is an extremely fast algorithm.

● Disadvantage:
■ Mergesort requires a second array as large as the original array.
12
 Quick sort

● Quick Sort: orders a list of values by partitioning the list

around one element called a pivot, then sorting each partition.

● Key Points:
■ choose one element in the list to be the pivot (= partition
element)
■ organize the elements so that all elements less than the
pivot are to its left and all greater are to its right
■ apply the quick sort algorithm (recursively) to both
partitions
13
 Quicksort Outline
● Divide and conquer approach
● Given array S to be sorted
• If size of S < 1 then done;
• Pick any element v in S as the pivot
• Partition S-{v} (remaining elements in S) into two groups
• S1 = {all elements in S-{v} that are smaller than v}
• S2 = {all elements in S-{v} that are larger than v}
• Return {quicksort(S1) followed by v followed by quicksort(S2)
}
● Trick lies in handling the partitioning (step 3).
■ Picking a good pivot
■ Efficiently partitioning in-place
14
 Quick sort illustrated
pick a pivot

40 18 37 2
10 32 6 35
12 17
partitio
6 17 n 40 37
10 12 2 18 32 35

quicksort quicksort
2 6 10 12 17 18 32 35 37 40

combine 15

2 6 10 12 17 18 32 35 37 40
Pseudocode

16
 Pseudocode

What is the running time of

partition()?
17
partition() runs in O(n) time
 Partitioning

● To partition a[left...right]:
1. Set p = a[left], l = left + 1, r = right;
2. while l < r, do
2.1. while l < right & a[l] < p, set l = l +
1
2.2. while r > left & a[r] >= p, set r = r
-1
2.3. if l < r, swap a[l] and a[r]
3. Set a[left] = a[r], a[r] = p 18
4. Terminate
 Example of partitioning

● choose pivot: 436924312189356

● search: 436924312189356
● swap: 433924312189656
● search: 433924312189656
● swap: 433124312989656
● search: 433124312989656
● swap: 433122314989656
● search: 4 3 3 1 2 2 3 1 4 9 8 9 6 5 6 (left > right)
● swap with pivot: 1 3 3 1 2 2 3 4 4 9 8 9 6 5 6
19
 Quicksort: Best Case Analysis

● Assume that keys are random, uniformly

distributed.
● What is best case running time?
● Best case: pivot is the median
■ Recursion:
1. Partition splits array in two sub-arrays of size n/2
2. Quicksort each sub-array
■ Depth of recursion tree? O(log2n)
■ Number of accesses in partition? O(n) 20
 Recursion Tree for Best Case
Partition Comparisons

Nodes contain problem size n n

n/2 n/2 n

n/4 n/4 n/4 n/4 n

n/8 n/8 n/8 n/8 n/8 n/8 n/8 n/8

.
.
.
.
.
.
>
n
. >
. >
>

T(n) = 2 T(n/2) + cn 21

T(n) = cn log n + n = O(n log n)

Quicksort: Average Case Analysis
● Assume that keys are random, uniformly
distributed.
● What is average case running time?
■ Many Recursions:
■ Depth of recursion tree? O(logxn)
○ x depends on how the partition is split
■ Number of accesses in partition? O(n)

22

O (n lg n)
 Quicksort: Worst Case Analysis
● Assume that keys are random, uniformly distributed.
● What is worst case running time?
● Pivot is the smallest (or largest) element all the time.
■ Recursion:
1. Partition splits array in three sub-arrays:
• one sub-array of size 0
• one sub-array with the pivot itself
• the other sub-array of size n-1
2. Quicksort each sub-array

■ Depth of recursion tree? O(n)

■ Number of accesses per partition? O(n)

23
 Worst Case Intuition
t(n) = n n
- -
0 1 n 1
n
- -
0 2 n 2
n
T(n) = T(n-1) + cn
- -
T(n-1) = T(n-2) + c(n-1)
0 3 n n
3
T(n-2) = T(n-3) + c(n-2) .
- .
.
-
…
4 4
T(2) = T(1) + 2c 0 1 1
24
0 0 0
 Picking the Pivot
● How would you pick one?

● Strategy 1: Pick the first element in S

■ Works only if input is random

■ What if input S is sorted, or even mostly sorted?

○ All the remaining elements would go into either S1 or S2!
○ Terrible performance!

■ Why worry about sorted input?

○ Remember 🡪 Quicksort is recursive, so sub-problems could be sorted
○ Plus mostly sorted input is quite frequent

25
 Picking the Pivot (contd.)
● Strategy 2: Pick the pivot randomly

■ Would usually work well, even for mostly sorted

input

■ Plus random number generation is an expensive

operation

26
 Picking the Pivot (contd.)
● Strategy 3: Median-of-three Partitioning

■ Ideally, the pivot should be the median of input array S

○ Median = element in the middle of the sorted sequence

■ Would divide the input into two almost equal partitions

■ Unfortunately, its hard to calculate median quickly, without sorting first!

■ So find the approximate median

○ Pivot = median of the left-most, right-most and center element of the array S
○ Solves the problem of sorted input

27
 Picking the Pivot (contd.)
● Example: Median-of-three Partitioning

■ Let input S = {6, 1, 4, 9, 0, 3, 5, 2, 7, 8}

■ left=0 and S[left] = 6

■ right=9 and S[right] = 8

■ center = (left+right)/2 = 4 and S[center] = 0

■ Pivot
○ = Median of S[left], S[right], and S[center]
○ = median of 6, 8, and 0
○ = S[left] = 6

28
 Partitioning Algorithm
● Original input : S = {6, 1, 4, 9, 0, 3, 5, 2, 7, 8}

● Get the pivot out of the way by swapping it with the last element

8 1 4 9 0 3 5 2 7 6
pivot
● Have two ‘iterators’ – i and j
■ i starts at first element and moves forward
■ j starts at last element and moves backwards

8 1 4 9 0 3 5 2 7 6
i j pivot
29
 Partitioning Algorithm (contd.)
❑ While (i < j)

■ Move i to the right till we find a number greater than pivot

■ Move j to the left till we find a number smaller than pivot

■ If (i < j) swap(S[i], S[j])

■ (The effect is to push larger elements to the right and smaller elements to the
left)

4. Swap the pivot with S[i]

30
 Partitioning Algorithm Illustrated
i j pivot
8 1 4 9 0 3 5 2 7 6
Move i j pivot
8 1 4 9 0 3 5 2 7 6
i j pivot
swap
2 1 4 9 0 3 5 8 7 6
i j pivot
move
2 1 4 9 0 3 5 8 7 6
i j pivot
swap
2 1 4 5 0 3 9 8 7 6
move j i pivot i and j
2 1 4 5 0 3 9 8 7 6 have
Swap S[i] crossed
2 1 4 5 0 3 6 8 7 9 31
with pivot
j i
pivot
 Dealing with small arrays
● For small arrays (N ≤ 20),
■ Insertion sort is faster than quicksort

● Quicksort is recursive
■ So it can spend a lot of time sorting small arrays

● Hybrid algorithm:
■ Switch to using insertion sort when problem size is small
(say for N < 20)

32
 Special cases
● What happens when the array contains many duplicate
elements?

● What happens when the array is already sorted (or nearly

sorted) to begin with?

33
 Quick sort: Final Comments
● If the array is sorted to begin with, Quicksort is terrible:
O(n2)
● However, Quicksort is usually O(n log2n)
● The constants are so good that Quicksort is generally the
fastest algorithm known
● Most real-world sorting is done by Quicksort
● For optimum efficiency, the pivot must be chosen
carefully
● “Median of three” is a good technique for choosing the
pivot
● However, no matter what you do, there will be some 34
cases where Quicksort runs in O(n2) time