3 Sorting Algorithms

Sorting

input: sequence (𝑎 (1) , … , 𝑎( 𝑛)) or set {𝑎 (1) , … , 𝑎 (𝑛) }

output: sequence (𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛))) with 𝑎 ( 𝜋 (1)) ≤ … ≤ 𝑎 ( 𝜋 (𝑛))

 Sorting

input: sequence (𝑎 (1) , … , 𝑎( 𝑛)) or set {𝑎 (1) , … , 𝑎 (𝑛) }

output: sequence (𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛))) with 𝑎 ( 𝜋 (1)) ≤ … ≤ 𝑎 ( 𝜋 (𝑛))

naive algorithm

1) compute minimum 𝑚𝑖𝑛 {𝑎 (1), … , 𝑎 (𝑛) }

 Sorting

input: sequence (𝑎 (1) , … , 𝑎( 𝑛)) or set {𝑎 (1) , … , 𝑎 (𝑛) }

output: sequence (𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛))) with 𝑎 ( 𝜋 (1)) ≤ … ≤ 𝑎 ( 𝜋 (𝑛))

naive algorithm

1) compute minimum 𝑚𝑖𝑛 {𝑎 (1), … , 𝑎 (𝑛) } ¿ 𝑚𝑖𝑛 {𝑚𝑖𝑛 {𝑎 (1) , 𝑎 (2) }, 𝑎 (3) , … , 𝑎 (𝑛)}

𝑀 (𝑛) comparisons
 Sorting

input: sequence (𝑎 (1) , … , 𝑎( 𝑛)) or set {𝑎 (1) , … , 𝑎 (𝑛) }

output: sequence (𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛))) with 𝑎 ( 𝜋 (1)) ≤ … ≤ 𝑎 ( 𝜋 (𝑛))

naive algorithm

1) compute minimum 𝑚𝑖𝑛 {𝑎 (1), … , 𝑎 (𝑛) } ¿ 𝑚𝑖𝑛 {𝑚𝑖𝑛 {𝑎 (1) , 𝑎 (2) }, 𝑎 (3) , … , 𝑎 (𝑛)}

𝑀 (𝑛) comparisons
 Sorting

input: sequence (𝑎 (1) , … , 𝑎( 𝑛)) or set {𝑎 (1) , … , 𝑎 (𝑛) }

output: sequence (𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛))) with 𝑎 ( 𝜋 (1)) ≤ … ≤ 𝑎 ( 𝜋 (𝑛))

naive algorithm

1) compute minimum 𝑚𝑖𝑛 {𝑎 (1), … , 𝑎 (𝑛) } ¿ 𝑚𝑖𝑛 {𝑚𝑖𝑛 {𝑎 (1) , 𝑎 (2) }, 𝑎 (3) , … , 𝑎 (𝑛)}

𝑀 (𝑛) comparisons
 Sorting

input: sequence (𝑎 (1) , … , 𝑎( 𝑛)) or set {𝑎 (1) , … , 𝑎 (𝑛) }

output: sequence (𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛))) with 𝑎 ( 𝜋 (1)) ≤ … ≤ 𝑎 ( 𝜋 (𝑛))

naive algorithm

1) compute minimum 𝑚𝑖𝑛 {𝑎 (1), … , 𝑎 (𝑛) } ¿ 𝑚𝑖𝑛 {𝑚𝑖𝑛 {𝑎 (1) , 𝑎 (2) }, 𝑎 (3) , … , 𝑎 (𝑛)}

comparisons

2) repeat
𝑠𝑜𝑟𝑡 ( 𝐴)=𝑚𝑖𝑛( 𝐴) ∘ 𝑠𝑜𝑟𝑡 ¿
 Sorting

input: sequence (𝑎 (1) , … , 𝑎( 𝑛)) or set {𝑎 (1) , … , 𝑎 (𝑛) }

output: sequence (𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛))) with 𝑎 ( 𝜋 (1)) ≤ … ≤ 𝑎 ( 𝜋 (𝑛))

naive algorithm

1) compute minimum 𝑚𝑖𝑛 {𝑎 (1), … , 𝑎 (𝑛) } ¿ 𝑚𝑖𝑛 {𝑚𝑖𝑛 {𝑎 (1) , 𝑎 (2) }, 𝑎 (3) , … , 𝑎 (𝑛)}

comparisons

2) repeat
𝑠𝑜𝑟𝑡 ( 𝐴)=𝑚𝑖𝑛( 𝐴) ∘ 𝑠𝑜𝑟𝑡 ¿

𝑆 (𝑛) comparisons
 Sorting

input: sequence (𝑎 (1) , … , 𝑎( 𝑛)) or set {𝑎 (1) , … , 𝑎 (𝑛) }

output: sequence (𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛))) with 𝑎 ( 𝜋 (1)) ≤ … ≤ 𝑎 ( 𝜋 (𝑛))

naive algorithm

1) compute minimum 𝑚𝑖𝑛 {𝑎 (1), … , 𝑎 (𝑛) } ¿ 𝑚𝑖𝑛 {𝑚𝑖𝑛 {𝑎 (1) , 𝑎 (2) }, 𝑎 (3) , … , 𝑎 (𝑛)}

comparisons

2) repeat
𝑠𝑜𝑟𝑡 ( 𝐴)=𝑚𝑖𝑛( 𝐴) ∘ 𝑠𝑜𝑟𝑡 ¿

𝑆 (𝑛) comparisons
 merge 2 sorted lists of length n/2 Merge Sort
input: sorted sequences

(𝑎 (1) , … , 𝑎( 𝑛/ 2)) 𝑎 (1) ≤ … ≤ 𝑎(𝑛 / 2)

with
(𝑏 (1)) , … , 𝑏(𝑛 /2) ¿
𝑏( 1) ≤ … ≤ 𝑏(𝑛 / 2)

output: merged sequence (𝑐 (1) , … ,𝑐 (𝑛)) sorted 𝑐 (1)≤ … ≤ 𝑐 (𝑛) ¿

 merge 2 sorted lists of length n/2 Merge Sort
input: sorted sequences

(𝑎 (1) , … , 𝑎( 𝑛)) 𝑎 (1) ≤ … ≤ 𝑎(𝑛)

with
(𝑏 (1)) , … , 𝑏(𝑚) ¿
𝑏( 1) ≤ … ≤ 𝑏(𝑛𝑚)

output: merged sequence (𝑐 (1) , … ,𝑐 (𝑛+ 𝑚)) sorted 𝑐 (1)≤ … ≤ 𝑐 (𝑛+ 𝑚) ¿

𝑛+𝑚 − 1 comparisons
 merge 2 sorted lists of length n/2 Merge Sort
input: sorted sequences

(𝑎 (1) , … , 𝑎( 𝑛)) 𝑎 (1) ≤ … ≤ 𝑎(𝑛)

with
(𝑏 (1)) , … , 𝑏(𝑚) ¿
𝑏( 1) ≤ … ≤ 𝑏(𝑛𝑚)

output: merged sequence (𝑐 (1) , … ,𝑐 (𝑛+ 𝑚)) sorted 𝑐 (1)≤ … ≤ 𝑐 (𝑛+ 𝑚) ¿

𝑛+𝑚 − 1 comparisons

𝑠𝑜𝑟𝑡 ¿
 merge 2 sorted lists of length n/2 Merge Sort
input: sorted sequences

(𝑎 (1) , … , 𝑎( 𝑛)) 𝑎 (1) ≤ … ≤ 𝑎(𝑛)

with
(𝑏 (1)) , … , 𝑏(𝑚) ¿
𝑏( 1) ≤ … ≤ 𝑏(𝑛𝑚)

output: merged sequence (𝑐 (1) , … ,𝑐 (𝑛+ 𝑚)) sorted 𝑐 (1)≤ … ≤ 𝑐 (𝑛+ 𝑚) ¿

𝑛+𝑚 − 1 comparisons

𝑠𝑜𝑟𝑡 ¿
difference equations (and difference inequalities)
 merge 2 sorted lists of length n/2 Merge Sort
input: sorted sequences

(𝑎 (1) , … , 𝑎( 𝑛)) 𝑎 (1) ≤ … ≤ 𝑎(𝑛)

with
(𝑏 (1)) , … , 𝑏(𝑚) ¿
𝑏( 1) ≤ … ≤ 𝑏(𝑛𝑚)

output: merged sequence (𝑐 (1) , … ,𝑐 (𝑛+ 𝑚)) sorted 𝑐 (1)≤ … ≤ 𝑐 (𝑛+ 𝑚) ¿

𝑛+𝑚 − 1 comparisons

𝑠𝑜𝑟𝑡 ¿

𝑆 (𝑛)<𝑛 ⋅ log 𝑛
can we sort asymptotically faster with comparisons?
 can we sort asymptotically faster with comparisons?

no

can represent any such algorithm as binary tree

• nodes: comparisons
 can we sort asymptotically faster with comparisons?

no
(𝑎 (1) , … , 𝑎( 𝑛))
can represent any such algorithm as binary tree
• nodes: comparisons
• at root: input sequence or set to be sorted
 can we sort asymptotically faster with comparisons?

no
(𝑎 (1) , … , 𝑎( 𝑛)) can represent any such algorithm as binary tree
• nodes: comparisons
• at root: input sequence or set to be sorted
• at leaves: sorted sequence
• one leaf for each permutation 𝜋

depth of tree = length of longest path = worst case number of comparisons

(𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛)))
 can we sort asymptotically faster with comparisons?

no
(𝑎 (1) , … , 𝑎( 𝑛)) can represent any such algorithm as binary tree
• nodes: comparisons
• at root: input sequence or set to be sorted
• at leaves: sorted sequence
• one leaf for each permutation 𝜋

𝑑 depth of tree = length of longest path = worst case number of comparisons

easy induction: binary tree with depth d has at most leaves

(𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛)))
 can we sort asymptotically faster with comparisons?

no
(𝑎 (1) , … , 𝑎( 𝑛)) can represent any such algorithm as binary tree
• nodes: comparisons
• at root: input sequence or set to be sorted
• at leaves: sorted sequence
• one leaf for each permutation 𝜋

𝑑 depth of tree = length of longest path = worst case number of comparisons

easy induction: binary tree with depth d has at most leaves

def: 𝑛! number of permutations of n numbers

(𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛)))
 can we sort asymptotically faster with comparisons?

no
(𝑎 (1) , … , 𝑎( 𝑛)) can represent any such algorithm as binary tree
• nodes: comparisons
• at root: input sequence or set to be sorted
• at leaves: sorted sequence
• one leaf for each permutation 𝜋

𝑑 depth of tree = length of longest path = worst case number of comparisons

easy induction: binary tree with depth d has at most leaves

def: 𝑛! number of permutations of n numbers

1 !=1 𝑛 !=𝑛 ⋅(𝑛 − 1) !

(𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛)))

choices of first element possible permutations of remaining elements

 can we sort asymptotically faster with comparisons?

no
(𝑎 (1) , … , 𝑎( 𝑛)) can represent any such algorithm as binary tree
• nodes: comparisons
• at root: input sequence or set to be sorted
• at leaves: sorted sequence
• one leaf for each permutation 𝜋

𝑑 depth of tree = length of longest path = worst case number of comparisons

easy induction: binary tree with depth d has at most leaves

def: 𝑛! number of permutations of n numbers

1 !=1 𝑛 !=𝑛 ⋅(𝑛 − 1) !

(𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛)))

choices of first element possible permutations of remaining elements

𝑑
2 ≥ 𝑛 !> 𝑛⋅ (𝑛 − 1 ) ⋅ … ⋅ 𝑛/ 2 >¿
 can we sort asymptotically faster with comparisons?

no
(𝑎 (1) , … , 𝑎( 𝑛)) can represent any such algorithm as binary tree
• nodes: comparisons
• at root: input sequence or set to be sorted
• at leaves: sorted sequence
• one leaf for each permutation 𝜋

𝑑 depth of tree = length of longest path = worst case number of comparisons

easy induction: binary tree with depth d has at most leaves

def: 𝑛! number of permutations of n numbers

1 !=1 𝑛 !=𝑛 ⋅(𝑛 − 1) !

(𝑎 ( 𝜋 (1)) , … , 𝑎 ( 𝜋 (𝑛)))

choices of first element possible permutations of remaining elements

𝑑
2 ≥ 𝑛 !> 𝑛⋅ (𝑛 − 1 ) ⋅ … ⋅ 𝑛/ 2 >¿

𝑑>(𝑛/ 2) ⋅ log (𝑛 / 2)
Probabilistic Analysis of Algorithms

based on ‚random events‘

whatever randomness is…

more in ‚theoretical computer science‘

or - time permitting - here
 Probabilistic Analysis of Algorithms

based on ‚random events‘

whatever randomness is…

more in ‚theoretical computer science‘

or - time permitting - here

two flavors
• random input
• modeling: our lack of understanding of what future inputs we will see
• bad: what if an adversary constructs the input - when it matters?

• the algorithm generates - and branches on - random data

• good: this is under the programmer’s control
• usual implementation: by a pseudo random number generator
 Probabilistic Analysis of Algorithms

based on ‚random events‘

whatever randomness is…

more in ‚theoretical computer science‘

or - time permitting - here

two flavors
• random input
• modeling: our lack of understanding of what future inputs we will see
• bad: what if an adversary constructs the input - when it matters?

• the algorithm generates - and branches on - random data

• good: this is under the programmer’s control
• usual implementation: by a pseudo random number generator
• generated data are completely predictable (run the generator)
• whatever randomness is… this is not it!
 Probabilistic Analysis of Algorithms

based on ‚random events‘

whatever randomness is…

more in ‚theoretical computer science‘

or - time permitting - here

two flavors
• random input
• modeling: our lack of understanding of what future inputs we will see
• bad: what if an adversary constructs the input - when it matters?

• the algorithm generates - and branches on - random data

• good: this is under the programmer’s control
• usual implementation: by a pseudo random number generator
• generated data are completely predictable (run the generator)
• whatever randomness is… this is not it!
the miracle

• you program with a pseudo random number generator

• you analyze as if the generated data was truly random
• the measured run time (usually) matches the analysis
 gentle reminder at this place:

mathematics is an experimental science

repairs just happen to be particularly rare

 gentle reminder at this place:

mathematics is an experimental science

repairs just happen to be particularly rare

1903: Russel paradox

 gentle reminder at this place:

mathematics is an experimental science

repairs just happen to be particularly rare

1903: Russel paradox

why does probabilistic modeling

of a completely deterministic -provably not random - algorithm
give the right results??

the miracle

• you program with a pseudo random number generator

• you analyze as if the generated data was truly random
• the measured run time (usually) matches the analysis
 Quicksort
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) }

 Quicksort
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) }

random experiment: choose ‚splitter‘ 𝑠 ∈{𝑎 (1), … , 𝑎 (𝑛) }

 Quicksort
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) }

random experiment: choose ‚splitter‘ 𝑠 ∈{𝑎 (1), … , 𝑎 (𝑛) }

all n splitters equally likely

 Quicksort
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) }

random experiment: choose ‚splitter‘ 𝑠 ∈{𝑎 (1), … , 𝑎 (𝑛) }

all n splitters equally likely

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 < 𝑠 }

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 > 𝑠 }

 Quicksort
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) } we assume here: a(i) mutually distinct

random experiment: choose ‚splitter‘ 𝑠 ∈{𝑎 (1), … , 𝑎 (𝑛) }

all n splitters equally likely

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 < 𝑠 }

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 > 𝑠 }

𝑠𝑜𝑟𝑡 ( 𝐴)=𝑠𝑜𝑟𝑡 ( 𝐴 ¿ ) ∘ 𝑠 ∘ 𝑠𝑜𝑟𝑡 ( 𝐴 ¿ )

 Quicksort. 1960, Tony Hoare
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) } we assume here: a(i) mutually distinct

random experiment: choose ‚splitter‘ 𝑠 ∈{𝑎 (1), … , 𝑎 (𝑛) }

all n splitters equally likely

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 < 𝑠 }

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 > 𝑠 }

𝑠𝑜𝑟𝑡 ( 𝐴)=𝑠𝑜𝑟𝑡 ( 𝐴 ¿ ) ∘ 𝑠 ∘ 𝑠𝑜𝑟𝑡 ( 𝐴 ¿ )

2013
 now computer science textbooks continue ….. since decades
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) } we assume here: a(i) mutually distinct

random experiment: choose ‚splitter‘ 𝑠 ∈{𝑎 (1), … , 𝑎 (𝑛) }

all n splitters equally likely

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 < 𝑠 }

n-1 comparisons
𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 > 𝑠 }

𝑠𝑜𝑟𝑡 ( 𝐴)=𝑠𝑜𝑟𝑡 ( 𝐴 ¿ ) ∘ 𝑠 ∘ 𝑠𝑜𝑟𝑡 ( 𝐴 ¿ )

𝑇 (𝑛) expected run time for sorting n elements

𝑛
𝑇 (𝑛)≤ 𝑛+(1/ 𝑛) ⋅ Σ 𝑖=1 (𝑇 (𝑖 − 1)+𝑇 (𝑛 − 𝑖))
 now computer science textbooks continue ….. since decades
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) } we assume here: a(i) mutually distinct

random experiment: choose ‚splitter‘ 𝑠 ∈{𝑎 (1), … , 𝑎 (𝑛) }

all n splitters equally likely

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 < 𝑠 }

n-1 comparisons
𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 > 𝑠 }

𝑠𝑜𝑟𝑡 ( 𝐴)=𝑠𝑜𝑟𝑡 ( 𝐴 ¿ ) ∘ 𝑠 ∘ 𝑠𝑜𝑟𝑡 ( 𝐴 ¿ )

𝑇 (𝑛) expected run time for sorting n elements in mathematics expected runtime has a definition

𝑛
𝑇 (𝑛)≤ 𝑛+(1/ 𝑛) ⋅ Σ 𝑖=1 (𝑇 (𝑖 − 1)+𝑇 (𝑛 − 𝑖))
 now computer science textbooks continue ….. since decades
here: the algorithm does the random experiments

input: (𝑎 (1) , … , 𝑎( 𝑛)) or set 𝐴= {𝑎 (1) ,… , 𝑎(𝑛) } we assume here: a(i) mutually distinct

random experiment: choose ‚splitter‘ 𝑠 ∈{𝑎 (1), … , 𝑎 (𝑛) }

all n splitters equally likely

𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 < 𝑠 }

n-1 comparisons
𝐴 ¿ ={𝑎 ∈ 𝐴∨𝑎 > 𝑠 }

𝑠𝑜𝑟𝑡 ( 𝐴)=𝑠𝑜𝑟𝑡 ( 𝐴 ¿ ) ∘ 𝑠 ∘ 𝑠𝑜𝑟𝑡 ( 𝐴 ¿ )

𝑇 (𝑛) expected run time for sorting n elements in mathematics expected runtime has a definition

𝑛
𝑇 (𝑛)≤ 𝑛+(1/ 𝑛) ⋅ Σ 𝑖=1 (𝑇 (𝑖 − 1)+𝑇 (𝑛 − 𝑖)) this should have a proof!