Skiena algorithm 2007 lecture09 linear sorting

Lecture 9:
Linear Sorting
Steven Skiena

Department of Computer Science
State University of New York
Stony Brook, NY 11794–4400

http://www.cs.sunysb.edu/∼skiena

Problem of the Day
The nuts and bolts problem is deﬁned as follows. You
are given a collection of n bolts of different widths, and n
corresponding nuts. You can test whether a given nut and bolt
together, from which you learn whether the nut is too large,
too small, or an exact match for the bolt. The differences in
size between pairs of nuts or bolts can be too small to see by
eye, so you cannot rely on comparing the sizes of two nuts or
two bolts directly. You are to match each bolt to each nut.

1. Give an O(n2 ) algorithm to solve the nuts and bolts
problem.
2. Suppose that instead of matching all of the nuts and bolts,
you wish to ﬁnd the smallest bolt and its corresponding
nut. Show that this can be done in only 2n − 2
comparisons.
3. Match the nuts and bolts in expected O(n log n) time.

Quicksort Pseudocode

Sort(A)
Quicksort(A,1,n)

Quicksort(A, low, high)
if (low < high)
pivot-location = Partition(A,low,high)
Quicksort(A,low, pivot-location - 1)
Quicksort(A, pivot-location+1, high)

Partition Implementation

Partition(A,low,high)
pivot = A[low]
leftwall = low
for i = low+1 to high
if (A[i] < pivot) then
leftwall = leftwall+1
swap(A[i],A[leftwall])
swap(A[low],A[leftwall])

Quicksort Animation

Q U I C K S O R T

Q I C K S O R T U

Q I C K O R S T U

I C K O Q R S T U

I C K O Q R S T U

I C K O Q R S T U

Best Case for Quicksort
Since each element ultimately ends up in the correct position,
the algorithm correctly sorts. But how long does it take?
The best case for divide-and-conquer algorithms comes when
we split the input as evenly as possible. Thus in the best case,
each subproblem is of size n/2.
The partition step on each subproblem is linear in its size.
Thus the total effort in partitioning the 2k problems of size
n/2k is O(n).

Best Case Recursion Tree

The total partitioning on each level is O(n), and it take
lg n levels of perfect partitions to get to single element
subproblems. When we are down to single elements, the
problems are sorted. Thus the total time in the best case is
O(n lg n).

Worst Case for Quicksort
Suppose instead our pivot element splits the array as
unequally as possible. Thus instead of n/2 elements in the
smaller half, we get zero, meaning that the pivot element is
the biggest or smallest element in the array.

Now we have n−1 levels, instead of lg n, for a worst case time
of Θ(n2 ), since the ﬁrst n/2 levels each have ≥ n/2 elements
to partition.
To justify its name, Quicksort had better be good in the
average case. Showing this requires some intricate analysis.
The divide and conquer principle applies to real life. If you
break a job into pieces, make the pieces of equal size!

Intuition: The Average Case for Quicksort
Suppose we pick the pivot element at random in an array of n
keys.

1 n/4 n/2 3n/4 n

Half the time, the pivot element will be from the center half
of the sorted array.
Whenever the pivot element is from positions n/4 to 3n/4, the
larger remaining subarray contains at most 3n/4 elements.

How Many Good Partitions
If we assume that the pivot element is always in this range,
what is the maximum number of partitions we need to get
from n elements down to 1 element?
(3/4)l · n = 1 −→ n = (4/3)l

lg n = l · lg(4/3)

Therefore l = lg(4/3) · lg(n) < 2 lg n good partitions sufﬁce.

How Many Bad Partitions?
How often when we pick an arbitrary element as pivot will it
generate a decent partition?
Since any number ranked between n/4 and 3n/4 would make
a decent pivot, we get one half the time on average.
If we need 2 lg n levels of decent partitions to ﬁnish the job,
and half of random partitions are decent, then on average the
recursion tree to quicksort the array has ≈ 4 lg n levels.

Since O(n) work is done partitioning on each level, the
average time is O(n lg n).

Average-Case Analysis of Quicksort (*)
To do a precise average-case analysis of quicksort, we
formulate a recurrence given the exact expected time T (n):
n 1
T (n) = (T (p − 1) + T (n − p)) + n − 1
p=1 n

Each possible pivot p is selected with equal probability. The
number of comparisons needed to do the partition is n − 1.
We will need one useful fact about the Harmonic numbers
Hn, namely
n
Hn = 1/i ≈ ln n
i=1

It is important to understand (1) where the recurrence relation

comes from and (2) how the log comes out from the
summation. The rest is just messy algebra.
n 1
T (n) = (T (p − 1) + T (n − p)) + n − 1
p=1 n
2 n
T (n) = T (p − 1) + n − 1
n p=1
n
nT (n) = 2 T (p − 1) + n(n − 1) multiply by n
p=1
n−1
(n−1)T (n−1) = 2 T (p−1)+(n−1)(n−2) apply to n-1
p=1
nT (n) − (n − 1)T (n − 1) = 2T (n − 1) + 2(n − 1)
rearranging the terms give us:
T (n) T (n − 1) 2(n − 1)
= +
n+1 n n(n + 1)

substituting an = A(n)/(n + 1) gives
2(n − 1) n 2(i − 1)
an = an−1 + =
n(n + 1) i=1 i(i + 1)
n 1
an ≈ 2 ≈ 2 ln n
i=1 (i + 1)
We are really interested in A(n), so
A(n) = (n + 1)an ≈ 2(n + 1) ln n ≈ 1.38n lg n

Pick a Better Pivot
Having the worst case occur when they are sorted or almost
sorted is very bad, since that is likely to be the case in certain
applications.
To eliminate this problem, pick a better pivot:
1. Use the middle element of the subarray as pivot.
2. Use a random element of the array as the pivot.
3. Perhaps best of all, take the median of three elements
(ﬁrst, last, middle) as the pivot. Why should we use
median instead of the mean?
Whichever of these three rules we use, the worst case remains
O(n2 ).

Is Quicksort really faster than Heapsort?

Since Heapsort is Θ(n lg n) and selection sort is Θ(n2 ), there
is no debate about which will be better for decent-sized ﬁles.
When Quicksort is implemented well, it is typically 2-3 times
faster than mergesort or heapsort.
The primary reason is that the operations in the innermost
loop are simpler.
Since the difference between the two programs will be limited
to a multiplicative constant factor, the details of how you
program each algorithm will make a big difference.

Randomized Quicksort
Suppose you are writing a sorting program, to run on data
given to you by your worst enemy. Quicksort is good on
average, but bad on certain worst-case instances.
If you used Quicksort, what kind of data would your enemy
give you to run it on? Exactly the worst-case instance, to
make you look bad.
But instead of picking the median of three or the ﬁrst element
as pivot, suppose you picked the pivot element at random.
Now your enemy cannot design a worst-case instance to give
to you, because no matter which data they give you, you
would have the same probability of picking a good pivot!

Randomized Guarantees
Randomization is a very important and useful idea. By either
picking a random pivot or scrambling the permutation before
sorting it, we can say:
“With high probability, randomized quicksort runs in
Θ(n lg n) time.”
Where before, all we could say is:
“If you give me random input data, quicksort runs in
expected Θ(n lg n) time.”

Importance of Randomization
Since the time bound how does not depend upon your input
distribution, this means that unless we are extremely unlucky
(as opposed to ill prepared or unpopular) we will certainly get
good performance.
Randomization is a general tool to improve algorithms with
bad worst-case but good average-case complexity.
The worst-case is still there, but we almost certainly won’t
see it.

Can we sort o(n lg n)?
Any comparison-based sorting program can be thought of as
deﬁning a decision tree of possible executions.
Running the same program twice on the same permutation
causes it to do exactly the same thing, but running it on
different permutations of the same data causes a different
sequence of comparisons to be made on each.

a1 < a2 ?
T F

a2 < a3 ? a1 < a3 ?

T F T F

(1,2,3) a1 < a3 ? (2,1,3) a2 < a3 ?
T F T F

(1,3,2) (3,1,2) (2,3,1) (3,2,1)

Claim: the height of this decision tree is the worst-case
complexity of sorting.

Lower Bound Analysis
Since any two different permutations of n elements requires
a different sequence of steps to sort, there must be at least n!
different paths from the root to leaves in the decision tree.
Thus there must be at least n! different leaves in this binary
tree.
Since a binary tree of height h has at most 2h leaves, we know
n! ≤ 2h, or h ≥ lg(n!).
By inspection n! > (n/2)n/2 , since the last n/2 terms of the
product are each greater than n/2. Thus

log(n!) > log((n/2)n/2 ) = n/2 log(n/2) → Θ(n log n)

Stirling’s Approximation
By Stirling’s approximation, a better bound is n! > (n/e)n
where e = 2.718.
h ≥ lg(n/e)n = n lg n − n lg e = Ω(n lg n)

Non-Comparison-Based Sorting
All the sorting algorithms we have seen assume binary
comparisons as the basic primative, questions of the form “is
x before y?”.
But how would you sort a deck of playing cards?
Most likely you would set up 13 piles and put all cards with
the same number in one pile.
With only a constant number of cards left in each pile, you can
use insertion sort to order by suite and concatenate everything
together.
If we could ﬁnd the correct pile for each card in constant
time, and each pile gets O(1) cards, this algorithm takes O(n)
time.

Bucketsort
Suppose we are sorting n numbers from 1 to m, where we
know the numbers are approximately uniformly distributed.
We can set up n buckets, each responsible for an interval of
m/n numbers from 1 to m
x x x x x x
x x

1 m/n m/n+1 2m/n 2m/n+1 3m/n ... ... m

Given an input number x, it belongs in bucket number
xn/m .
If we use an array of buckets, each item gets mapped to the
right bucket in O(1) time.

Bucketsort Analysis
With uniformly distributed keys, the expected number of
items per bucket is 1. Thus sorting each bucket takes O(1)
time!
The total effort of bucketing, sorting buckets, and concatenat-
ing the sorted buckets together is O(n).
What happened to our Ω(n lg n) lower bound!

Worst-Case vs. Assumed-Case
Bad things happen to bucketsort when we assume the wrong
distribution.
xx
xxx x
x xx x x
x
x x x
xx x
1 m/n m/n+1 2m/n 2m/n+1 3m/n ... ... m

We might spend linear time distributing our items into
buckets and learn nothing.
Problems like this are why we worry about the worst-case
performance of algorithms!

Real World Distributions
The worst case “shouldn’t” happen if we understand the
distribution of our data.
Consider the distribution of names in a telephone book.
• Will there be a lot of Skiena’s?
• Will there be a lot of Smith’s?
• Will there be a lot of Shifﬂett’s?
Either make sure you understand your data, or use a good
worst-case or randomized algorithm!

The Shifﬂett’s of Charlottesville
For comparison, note that there are seven Shifﬂett’s (of
various spellings) in the 1000 page Manhattan telephone
directory.

Skiena algorithm 2007 lecture09 linear sorting

Recommended

More Related Content

What's hot (20)

Similar to Skiena algorithm 2007 lecture09 linear sorting (20)

More from zukun (20)

Recently uploaded (20)

Skiena algorithm 2007 lecture09 linear sorting