Sorting Algorithms

Lecturer: Dr. Keyhanipour

 Outline
• Iterative sorting algorithms (comparison based)
• Selection Sort
• Bubble Sort
• Insertion Sort
• Recursive sorting algorithms (comparison based)
• Merge Sort
• Quick Sort
• Heap Sort
• Radix sort (non-comparison based)

2
 Sorting
• Sorting refers to operations of arranging a set of data in a given
order.
• Rearrange data[0], data[1], …, data[n-1] into ascending (or
descending) order. When done:
𝑑𝑎𝑡𝑎[0] <= 𝑑𝑎𝑡𝑎[1] <= … <= 𝑑𝑎𝑡𝑎[𝑛 − 1]

• We only consider sorting data in ascending order

• Example: 8, 6, 9, 4, 3 ⇒ 3, 4, 6, 8, 9

3
 Why Study Sorting?
• When an input is sorted, many problems become easy (e.g.
searching, min, max, k-th smallest)
• Sorting has a variety of interesting algorithmic solutions that
embody many ideas:
• Comparison vs. non-comparison based
• Iterative
• Recursive
• Divide-and-conquer
• Randomized algorithms
• Best/worst/average-case bounds

4
 Applications of Sorting
• Efficient searching
• Uniqueness testing
• Deleting duplicates
• Prioritizing events
• Frequency counting
• Set intersection/union
• Finding a target pair 𝑥, 𝑦 such that: 𝑥 + 𝑦 = 𝑧

5
 Basic Types of Sorting
• Internal Sorting:
• If all the data to be sorted can be adjusted in main memory then it is
called as Internal Sorting.
• External Sorting:
• If data to be stored is large and acquires external memory then the
type is called as External Sorting.

6
 Simple Sorting Algorithms
• Selection Sort
• Bubble Sort
• Insertion Sort
• Merge Sort

7
Selection Sort

8
 Selection Sort
• A simple sorting algorithm
• An in-place and comparison based algorithm, in which:
• The list is divided into two parts:
• the sorted part at the left end, and
• the unsorted part at the right end.
• Initially, the sorted part is empty and the unsorted part is the
entire list.

9
 Selection Sort: Description
• The smallest element is selected from the unsorted array and
swapped with the leftmost element, and that element becomes
a part of the sorted array.
• This process continues moving unsorted array boundary by
one element to the right
• Each time selecting an item according to its ordering and
placing it in the correct position in the sequence.

10
 The Selection Sort Algorithm
• For each index position 𝑖:
• Find the smallest data value in the array from positions 𝑖
through “𝑙𝑒𝑛𝑔𝑡ℎ − 1”, where length is the number of data values
stored.
• Exchange (swap) the smallest value with the value at position 𝑖.

11
Selection Sort: Animated Illustration

12
 Selection Sort: Pseudocode
template<class T>
void selectionSort(T data[], int n) {
int j;
for (int i = 0; i < n-1; i++) {
// find the index of the smallest element in the unsorted part of array
for (j = i+1, least = i; j < n; j++)
if (data[j] < data[least])
least = j;
// put the ith smallest to the i-1th position
swap(data[least],data[i]);
}
}

13
Selection Sort: Example 1

14
Selection Sort: Example 2

15
Selection Sort: Example 3

16
 Selection Sort: An alternative

• Find the greatest value in the array,

swap it into the rightmost component,
and then forget the rightmost
component.
• Do this repeatedly.

17
 Selection Sort: Analysis
• Time complexity:
• Worst case performance: 𝑂(𝑛2 )
• Best case performance: 𝑂(𝑛2 )
• Average case performance: 𝑂(𝑛2 )

• Space complexity:
• Worst case space complexity: 𝑂(𝑛) total, 𝑂(1) auxiliary

18
Bubble Sort

19
 Bubble Sort: Algorithm
• Algorithm of bubble sort includes two steps repeated until the
list is sorted:
• Compare adjacent elements, if the element on right side is smaller,
then swap their positions.
• Compare first element, second element and so on. After the
completion of Pass 1, the largest element is at last position.
• Invariant: each pass guarantees that largest remaining element
is in the correct position.

20
Bubble Sort: Illustration

21
Bubble Sort: Animated Illustration

22
 Bubble Sort: Pseudocode
template<class T>
void bubbleSort(T data[], int n) {
for (int i = n-1; i >=1; i--) // n passes, each pass ”bubble” one element
// keep comparing adjacent elements

for (int j = 1; j <= i; j++)

// Swap if the items are out of order
if (data[j] < data[j-1])
swap(data[j], data[j-1]);
}

23
 Bubble Sort: Example 1
Start – unsorted

• Compare, swap (0, 1)

• Compare, swap (1, 2)

• Compare, no swap

• Compare, no swap

• Compare, swap (4, 5)

• 99 in position
24
 Bubble Sort: Example 1 (cont.)

Pass 2

• swap (0, 1)

• no swap

• no swap

• swap (3, 4)

• 21 in position

25
 Bubble Sort: Example 1 (cont.)
Pass 3

• no swap

• no swap

• swap (2, 3)

• 12 in position, Pass 4

• no swap

• swap (1, 2)

• 8 in position, Pass 5

• swap (1, 2)

• Done
26
Bubble Sort: Example 2

27
 Bubble Sort: Analysis
• Time complexity:
• Worst case performance: 𝑂(𝑛2 )
• Best case performance: 𝑂(𝑛2 )
• Average case performance: 𝑂(𝑛2 )

• Space complexity:
• Worst case space complexity: 𝑂(𝑛) total, 𝑂(1) auxiliary

28
Insertion Sort

29
 Insertion Sort: Algorithm
• In insertion sort, the elements are compared and inserted to the
respective index place.
• It starts with comparison of 1st and 0th element in pass 1.
• In pass 2 the second element is compared with the 1st and 0th
elements.
• Doing so with all the elements in the list appropriate element is
inserted by shifting elements on right.

30
Insertion Sort: Illustration

31
Insertion Sort: Animated Illustration

32
 Insertion Sort: Pseudocode
template<class T>
void insertionSort(T data[], int n) {
for (int i = 1; i < n; i++) {
int next = data[i]; // next is the item to be inserted
// Shift sorted items to make place for next
for (int j = i-1; j >= 0 && data[j] > next; j--)
data[j+1] = data[j];
data[j+1] = next; // Insert next to the correct location
}
}

33
Insertion Sort: Example 1

34
Insertion Sort: Example 2

35
 Insertion Sort: Example 2 (cont.)
• Fifth Pass of the Insertion Sort in detail

36
 Insertion Sort: Analysis
• Time complexity:
• Worst case performance: 𝑂(𝑛2 )
• Best case performance: 𝑂(𝑛)
• Average case performance: 𝑂(𝑛2 )

• Space complexity:
• Worst case space complexity: 𝑂(𝑛) total, 𝑂(1) auxiliary

37
 Outline
• Iterative sorting algorithms (comparison based)
• Selection Sort
• Bubble Sort
• Insertion Sort
• Recursive sorting algorithms (comparison based)
• Merge Sort
• Quick Sort
• Heap Sort
• Radix sort (non-comparison based)

38
Merge Sort

39
 Merge Sort
• Merge Sort is a divide-and-conquer sorting algorithm
• Merge sort first divides the array into equal halves and then
combines them in a sorted manner.
• Divide step:
• Divide the array into two (equal) halves
• Recursively sort the two halves
• Conquer step:
• Merge the two halves to form a sorted array

40
Merge Sort: Illustration

41
Merge Sort: Merge

42
Merge Sort: Animated Illustration

43
 Merge Sort: Pseudocode
template<class T>
void mergeSort(T data[], int low, int high) {
if (low < high) {
int mid = (low+high) / 2;
mergeSort(data, low , mid);
mergeSort(data, mid+1, high);
merge(data, low, mid, high);
}
}

44
 Merge Sort: Pseudocode (cont.)
template<class T>
void merge(T a[], int low, int mid, int high) {
int n = high-low+1;
int* b = new int[n]; b is a temporary array to store result
int left=low, right=mid+1, bIdx=0;
while (left <= mid && right <= high) {
if (a[left] <= a[right])
b[bIdx++] = a[left++]; Normal Merging Where both
else halves have unmerged items
b[bIdx++] = a[right++];
}
while (left <= mid) b[bIdx++] = a[left++]; Remaining items
while (right <= high) b[bIdx++] = a[right++]; are copied into b[]
for (int k = 0; k < n; k++) Remaining items
a[low+k] = b[k]; are copied into b[]
delete [] b;
} 45
 Merge Sort: Example
• Splitting the list in the Merge sort

46
 Merge Sort: Example (cont.)
• Lists as they are merged together

47
 Merge Sort: Analysis
• Time complexity:
• Worst case performance: 𝑂(𝑛 log 𝑛)
• Best case performance: 𝑂(𝑛 log 𝑛) typical
• Average case performance: 𝑂(𝑛 log 𝑛)

• Space complexity:
• Worst case space complexity: 𝑂(𝑛) total, 𝑂(𝑛) auxiliary

48
Radix Sort (non-comparison based)

49
 Radix Sort: Idea
• Treats each data to be sorted as a character string
• It is not using comparison, i.e. no comparison between the
data is needed
• In each iteration:
• Organize the data into groups according to the next character in each
data
• The groups are then “concatenated” for next iteration

50
Radix Sort: Example 1

51
 Types of Radix Sort
• Two classifications of radix sorts are:
• The Least Significant Digit (LSD) radix sorts, and
• The Most Significant Digit (MSD) radix sorts.
• LSD radix sorts process the integer representations starting
from the least digit and move towards the most significant
digit.
• MSD radix sorts work the other way around.

52
 Radix Sort: Pseudocode
template<class T>
void radixSort(T data[], int d) {
int i;
int power = 1; 10 groups. Each is a queue to retain the
queue<int> digitQueue[10]; order of items
for (i = 0; i < d; i++) {
• distribute(): Organize all items in v into
distribute(v, digitQueue, power); groups using digit indicated by the power
collect(digitQueue, v); • collect(): Place items from the groups
power *= 10; back into v, i.e. “concatenate” the groups
}
}

53
Radix Sort: Example 2

54
 Radix Sort: Example 3

Unsorted Sort digits Sort tens Sort hundreds

235 162 628 162
162 734 734 175
734 674 235 235
175 235 237 237
237 175 162 628
674 237 674 674
628 628 175 734

55
 Radix Sort: Analysis
• For each iteration:
• Pass over 𝑛 d-digit number to place them into group
• Then go through them again to concatenate the groups
• Time Complexity of each iteration is 𝑂(𝑛)
• Number of iterations is 𝑑, the maximum number of digits (or
maximum number of characters)
• Total Time Complexity is thus 𝑂(𝑑𝑛)
• Space complexity:
• Worst case space complexity: 𝑂(𝑛 + 𝑑) total

56
Summary

57
 Summary

Space
Time Complexity
Sorting Complexity
Algorithm Average
Best Case Worst Case Worst Case
Case
Selection Sort Ω(𝑛2 ) Θ(𝑛2 ) 𝑂(𝑛2 ) 𝑂(1)
Bubble Sort Ω(𝑛) Θ(𝑛2 ) 𝑂(𝑛2 ) 𝑂(1)
Insertion Sort Ω(𝑛) Θ(𝑛2 ) 𝑂(𝑛2 ) 𝑂(1)
Merge Sort Ω(𝑛 log 𝑛) Θ(𝑛 log 𝑛) 𝑂(𝑛 log 𝑛) 𝑂(𝑛)
Radix Sort Ω(𝑛𝑑) Θ(𝑛𝑑) 𝑂(𝑛𝑑) 𝑂(𝑛 + 𝑑)
58
 Outline
• Iterative sorting algorithms (comparison based)
• Selection Sort
• Bubble Sort
• Insertion Sort
• Recursive sorting algorithms (comparison based)
• Quick Sort
• Merge Sort
• Heap Sort
• Radix sort (non-comparison based)

59