CS214 – Data Structures

Lecture 04&5: Quadratic Sorting

Slides by
Dr-Mohamed El-Ramly, PhD
Dr-Basheer Youssef
 Agenda
0. Why Sorting?
1. Quadratic Sorting algorithms
• Selection Sort
• Insertion Sort
• Bubble Sort
2. Theoretical Boundaries of Sorting Problem
3. STL support for sorting
4. Sub-quadratic Sorting
• Merge Sort
• Shell Sort
• Quick Sort
• Heap Sort
2
 Chapter Objectives

• To learn how to use the standard sorting

algorithms in STL
• To learn how to implement the following sorting
algorithms: selection sort, bubble sort, insertion
sort, shell sort, merge sort, heap sort and quick.
• To understand the difference in performance of
these algorithms, and which to use for small,
medium and large sets of data and for what
types of data.

Chapter 9: Sorting 3
 Why Sorting?

• The efficiency of data handling can be

substantially increased if the data is sorted.
• Imagine an unsorted phone directory or a
dictionary!
• Often, it is required to sort data before
processing.
• First, we need to decide on the sorting criterion
• Second we need to decide on the sorting
algorithm

Chapter 9: Sorting 4
 Which Sorting Algorithms?
• To compare algorithm efficiency, we need a
machine-independent criterion for evaluating
their performance.
• Two factors determine the algorithm
performance:
• The number of comparisons
• The number of movements
• We use Big-O notation for this purpose.

Chapter 9: Sorting 5
 What Affects Performance?
• Initial Order of the Data
• Does the algorithm recognize already ordered
data?
• How much time is spent in on ordering already
ordered data?
• Algorithm’s intelligence
• Best / Worst / Average cases
• Comparisons / Movements: simple or complex?
• Do we compare int values or arrays?
• Do we move int values or structures?
Chapter 9: Sorting 6
Chapter 9: Sorting 7
 The family of sorting methods

Main sorting themes

Address-
Comparison-based -based
sorting sorting

Transposition count
sorting Sort

BubbleSort

Insert and Diminishing Divide and

keep sorted increment conquer
sorting

QuickSort MergeSort
Insertion
sort ShellSort

8
 Insertion Sort
• Based on the technique used by card players to arrange
a hand of cards
• Player keeps the cards that have been picked up so
far in sorted order
• When the player picks up a new card, he makes room
for the new card and then inserts it in its proper place

Chapter 9: Sorting 9
 Insertion Sort Algorithm

• For each array element from the second to the last

(nextPos = 1)
• Insert the element at nextPos where it belongs in the
array, increasing the length of the sorted subarray by
1

Chapter 9: Sorting 10
 One step of insertion sort

next to be
sorte inserted
d
3 4 7 12 14 14 20 21 33 38 10 55 9 23 28 16
temp
less 10
than 10
3 4 7 10 12 14 20
12 14 14 21 21 38
20 33 33 10
38 55 9 23 28 16

sorted

Chapter 9: Sorting 11
 Psuedocode for Insertion Sort

• insertionSort (data [], n)

for (i = 1; i < n; i++)

move all elements data [j] > data [i] by 1 position;

place data [i] in its proper position;

Chapter 9: Sorting 12
 C++ Code for Insertion Sort
• template <class T>

void insertionSort (T data[], int n) {

for (int i = 1, j; i < n; i++)
T tmp = data [i];

for (j = i; j > 0 && tmp < data [j-1];

j--)
data [j] = data [j – 1];
data [j] = tmp;
}

Chapter 9: Sorting 13
 Analysis of insertion sort
• Insertion sort sorts array when really needed.
• If the array is already sorted, moves are very few.
• On the other hand, shifting can be time consuming.

Chapter 9: Sorting 14
 Best Case
• Outer loop n – 1
• T tmp = data [i]; 1 move x n – 1 times
• data [j] = tmp; 1 move x n – 1 times
• tmp < data [j-1] 1 comparison x n – 1

• Moves: 2 (n - 1)
• Comparisons: n – 1

• O (n)

Chapter 9: Sorting 15
 Worst Case
• Happens when the data is in reverse order
• Outer loop n – 1
• T tmp = data [i]; 1 move x n – 1 times
• data [j] = tmp; 1 move x n – 1 times

• Inner loop executes, 1, 2, …, n – 1

• For each inner loop, there is 1 comparison
• For each inner loop, there is 1 move

Chapter 9: Sorting 16
 Worst Case
• Comparisons: (1 + 2 + …. + (n - 1)) = n (n – 1) / 2
 O(n2)

• Moves: (1 + 2 + …. + (n - 1)) + 2 (n – 1)
= n (n – 1) / 2 + 2 (n – 1)
= n2/2 + 3n/2 – 2
 O(n2)

Chapter 9: Sorting 17
 Average Case
• This is an approximate analysis simpler than the book
• Assume the data is randomly ordered
• Outer loop n – 1
• T tmp = data [i]; 1 move x n – 1 times
• data [j] = tmp; 1 move x n – 1 times

• Inner loop executes, 1, 2, …, n – 1

• For each inner loop, there is on average i /2 comparison
• For each inner loop, there is on average i /2 moves

Chapter 9: Sorting 18
 Average Case
• Comparisons: (1 + 2 + …. + (n - 1))/2 = n (n – 1) / 4
 O(n2)

• Moves: (1 + 2 + …. + (n - 1))/2 + 2 (n – 1)
= n (n – 1) / 4 + 2 (n – 1)
= n2/4 + 7n/4 – 2
 O(n2)

Chapter 9: Sorting 19
 Analysis of Insertion Sort
• Maximum number of comparisons is O(n*n)
• In the best case, number of comparisons is O(n)
• The number of shifts performed during an insertion is
one less than the number of comparisons or, when the
new value is the smallest so far, the same as the number
of comparisons
• A shift in an insertion sort requires the movement of only
one item whereas in a bubble or selection sort an
exchange involves a temporary item and requires the
movement of three items

Chapter 9: Sorting 20
Chapter 9: Sorting 21
 Selection Sort
• Selection sort is a relatively easy to understand
algorithm
• Sorts an array by making several passes through the
array, selecting the next smallest item in the array each
time and placing it where it belongs in the array
• It attempts to localize exchanges by putting the item
directly in its final place.
• Efficiency is O(n*n)

Chapter 9: Sorting 22
 Selection Sort
• Given an array of length n,
• Search elements 0 through n-1 and select the smallest
• Swap it with the element in location 0
• Search elements 1 through n-1 and select the smallest
• Swap it with the element in location 1
• Search elements 2 through n-1 and select the smallest
• Swap it with the element in location 2
• Search elements 3 through n-1 and select the smallest
• Swap it with the element in location 3
• Continue in this fashion until there’s nothing left to search

Chapter 9: Sorting 23
Example and analysis of Selection
Sort
• The Selection Sort might swap an array
7 2 8 5 4 element with itself--this is harmless,
and not worth checking for
2 7 8 5 4 • Analysis:
• The outer loop executes n-1 times
2 4 8 5 7 • The inner loop executes about n/2
times on average (from n to 2 times)
2 4 5 8 7 • Work done in the inner loop is
constant (swap two array elements)
2 4 5 7 8 • Time required is roughly (n-1)*(n/2)
• You should recognize this as O(n2)
Chapter 9: Sorting 24
 Psuedocode for Selection Sort

• selectionSort (data [], n)

for (i = 1; i < n-1; i++)

select smallest element from data [i],…,data [n-1];

swap it with data [i];

Chapter 9: Sorting 25
 C++ Code of Selection Sort
• template <class T>

void selectionSort(T data[], int n) {

for (int i = 0, j, least; i < n-1; i++) {

for (j = i+1, least = i; j < n; j++)

if (data [j] < data [least])

least = j;
swap (data [least], data [i]);
}
}

Chapter 9: Sorting 26
 Analysis of Selection Sort
• template <class T>

void selectionSort(T data[], int n) {

for (int i = 0, j, least; i < n-1; i++) {
n-1

for (j = i+1, least = i; j < n; j++)

n-1, n-2,…,1

if (data [j] < data [least]) 1 comparison

least = j;
swap (data [least], data [i]); 1 swap
}
}

Chapter 9: Sorting 27
 Best / Worst / Average Case
• It does not recognize order of data
• Comparisons: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2

 O (n )2

• Swaps: n – 1

 O (n)
• Moves: 3 (n – 1)

 O (n)
Chapter 9: Sorting 28
 Analysis of Selection Sort
• What conclusion can we make about Selection Sort?
• How intelligent is Selection Sort?
• What do you notice about the number of swaps?
• Modify the algorithm to further reduce the number of
swaps.

Chapter 9: Sorting 29
Chapter 9: Sorting 30
 Bubble Sort
• Compares adjacent array elements and exchanges their
values if they are out of order
• Smaller values bubble up to the top of the array and
larger values sink to the bottom

Chapter 9: Sorting 31
 Example of Bubble Sort

7 2 8 5 4 2 7 5 4 8 2 5 4 7 8 2 4 5 7 8
2 7 8 5 4 2 7 5 4 8 2 5 4 7 8 2 4 5 7 8
2 7 8 5 4 2 5 7 4 8 2 4 5 7 8 (don
e)
2 7 5 8 4 2 5 4 7 8

2 7 5 4 8

Chapter 9: Sorting 32
 Psuedocode for Bottom-up
Bubble Sort
• bubbleSort (data [], n)

for (i = 1; i < n-1; i++)

for (j = n - 1; j > i; --j)

swap items j and j – 1 if out of

order;

Chapter 9: Sorting 33
 C++ Code of Bottom-up Bubble
Sort
• template <class T>

void bubbleSort(T data[], int n) {

for (int i = 0; i < n - 1; i++) {

for (int j = n - 1; j > i; --j)

if (data [j] < data [j-1])

swap (data [j], data [j - 1]);

}

Chapter 9: Sorting 34
 Analysis of Bubble Sort
• template <class T>

void bubbleSort(T data[], int n) {

for (int i = 0; i < n - 1; i++) { n-1

for (int j = n - 1; j > i; --j) n-1,…,1

if (data [j] < data [j-1]) 1 comparison

swap (data [j], data [j - 1]);

} 1 swap

Chapter 9: Sorting 35
 Worst Case
• Reverse ordered data
• Comparisons: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2

 O (n ) 2

• Swaps: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2

 O (n ) 2

• Moves: 3 n (n – 1) / 2

 O (n ) 2

Chapter 9: Sorting 36
 Best Case
• When array is ordered
• Comparisons: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2

 O (n )2
 Can it be improved to O(n)

• Swaps: none

 O (1)
• Moves: none

 O (1)
Chapter 9: Sorting 37
 Average Case
• Comparisons: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2

 O (n ) 2

• Swaps: 1/2 + 2/2 + . . . + (n - 1)/2 = n (n - 1) / 4

 O (n )2

• Moves: 3 n (n – 1) / 4

 O (n )2

Chapter 9: Sorting 38
 Analysis of Bubble Sort
• Provides excellent performance in some cases and very
poor performances in other cases
• Main problem: it moves data up, one step at a time
• Works best when array is nearly sorted to begin with
• Worst case number of comparisons is O(n2)
• Worst case number of exchanges is O(n2)
• Best case occurs when the array is already sorted
• O(n) comparisons
• O(1) exchanges

Chapter 9: Sorting 39
Chapter 9: Sorting 40
Comparison of Quadratic Sorts
• None of the algorithms are particularly good for large
arrays

Chapter 9: Sorting 41
Chapter 9: Sorting 42
Chapter 9: Sorting 43
 Comparison of Quadratic Sorts
• Bubble Sort, Selection Sort, and Insertion Sort are all O(n2)
• As we will see later, we can do much better than this with
somewhat more complicated sorting algorithms
• Within O(n2),
• Bubble Sort is very slow, and should probably never be
used for anything
• Selection Sort is intermediate in speed
• Insertion Sort is usually the fastest of the three--in fact, for
small arrays (say, 10 or 15 elements), insertion sort is
faster than more complicated sorting algorithms
• Selection Sort and Insertion Sort are “good enough” for small
arrays

Chapter 9: Sorting 44
Chapter 9: Sorting 45
 Theoretical Analysis of Sorting
Problem

• Quadratic algorithms are not efficient ?

• Can we obtain better efficiency ?
• We need a lower bound or best performance.
• We focus on comparisons not moves because in ideal
cases, you can avoid any moves (e.g., bubble sort)

• What is the best estimate of the number of item

comparisons if the array is randomly ordered?

Chapter 9: Sorting 46
Chapter 9: Sorting 47
Chapter 9: Sorting 48
Chapter 9: Sorting 49
 Theoretical Analysis of Sorting
Problem

• Every sorting algorithms can be represented by a

decision tree.
• n – elements array may have n! ways of ordering
• But leaves may be more due to failures or repetitions

Chapter 9: Sorting 50
Chapter 9: Sorting 51
 Theoretical Analysis of Sorting
Problem
• An i-level complete decision tree has 2i-1 leaves and 2i-1 -1
non-terminals and 2i -1 total nodes.

• So, all non-complete trees with the same number of i levels

have fewer nodes, i.e., k+m  2i -1 (m leaves, k nonterminals) and k 
2i-1 -1 and m  2i-1

• n! are the possible ways of ordering an n-array

• n!  m  2i-1
• log (n!) + 1  i (We assume i is the lowest level referring to max
number of comparisons needed and longest path on the tree)

Chapter 9: Sorting 52
 Theoretical Analysis of Sorting
Problem

• i  log (n!) + 1
• The path length of a tree with n! nodes should be at least
log (n!)
• In other words O (log (n!)) is the best to expect in the
worst case for the number of comparisons.

• The same applies to average case.

Chapter 9: Sorting 53
 Theoretical Analysis of Sorting
Problem
• log (n!) is the min number of comparisons needed
• n! = n x (n - 1) x (n - 2) x … x 1
•  n x n x n x ... x n
•  nn
• log (n!)  log (n n)
• log (n!)  n log (n)

• log (n!)  O (n log (n ))

• Also, it can be proven that log (n!)   (n log (n ))

• Hence we should aim for a better sorting algorithm
Chapter 9: Sorting 54
Comparison of Average Case

2 times better

Huge improvement is still

possible, theoretically

Chapter 9: Sorting 55