CMSC 351: Big Notation

Justin Wyss-Gallifent

May 25, 2023

1 Inspiration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 The Bigs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1 Big-O Notation . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Big-Omega and Big-Theta Notations . . . . . . . . . . . . 4
2.3 All Together . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 A Limit Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4 Common Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5 Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
6 Additional Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.1 Use of n vs x . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.2 Cautious Comparisons . . . . . . . . . . . . . . . . . . . . 10
7 Thoughts, Problems, Ideas . . . . . . . . . . . . . . . . . . . . . . 11

1
1 Inspiration
Suppose two algorithms do exactly the same thing to lists of length n. We find
out that the time they take in seconds is as follows. Note that these are just
made up!
n A1 (n) A2 (n)
10 6 1
20 12 6
30 18 17
40 24 25
50 28 40
60 30 63
70 38 82
80 45 109
90 50 140
100 59 190
Observe that Algorithm 2 is better (faster) up until about n = 40, and then
Algorithm 1 is better.

But can we formalize this more, both the comparison and the values themselves?

It turns out that the values satisfy:

0.4n ≤ A1 (n) ≤ 0.6n

and:
0.01n2 ≤ A2 (n) ≤ 0.02n2

Although we don’t have an exact knowledge about other values we do certainly

have a more rigorous way not only of comparing the two algorithms but of
understandng each algorithm independently.

For example we can say that if n = 150 then Algorithm 2 will take at most
0.02(150)2 = 450 seconds. In this case we have an upper bound which is a
multiple of n2 .

Our goal is to formalize these notions.

2 The Bigs
2.1 Big-O Notation
Recall the definition:
Definition 2.1.1. We say that:
f (x) = O(g(x)) if ∃x0 , C > 0 such that ∀x ≥ x0 , f (x) ≤ Cg(x)

2
We think of this as stating that eventually f (x) is smaller than some constant
multiple of g(x).

Example 2.1. For example, here f (x) = O(x2 ) with C = 2 and x0 as

shown:

Cg(x) = 2x2

f (x)

x0

Note 2.1.1. There’s frequently (but not always) a trade-off in that if C is large
then x0 might be smaller, or vice-versa. In light of this note that “eventually”
could mean for a very large x0 .

Example 2.2. It’s true that 42000x lg x = O(x2 ) with C = 10 because

eventually 42000x lg x ≤ 10x2 . However “eventually” in this case means
x0 ≈ 67367. In other words this is the smallest x0 such that if x ≥ x0 then
42000x lg x ≤ 10x2 .

3
2.2 Big-Omega and Big-Theta Notations
We can extend upon this with:
Definition 2.2.1. We have:
f (x) = Ω(g(x)) if ∃x0 , B > 0 such that ∀x ≥ x0 , f (x) ≥ Bg(x)
1
Example 2.3. For example, here f (x) = Ω(x2 ) with B = 2 and x0 as
shown:

f (x)

Bg(x) = 21 x2

x0

and with:
Definition 2.2.2. We have:
f (x) = Θ(g(x)) if ∃x0 , B > 0, C > 0 such that ∀x ≥ x0 , Bg(x) ≤ f (x) ≤ Cg(x)
1
Example 2.4. For example, here f (x) = Θ(x2 ) with B = 2 and C = 2 and
x0 as shown:

Cg(x) = 2x2

f (x)

Bg(x) = 21 x2

x0

4
2.3 All Together
The basic idea is that O provides an upper bound for f (x), Ω provides a lower
bound and Θ provides a tight bound. Therefore f (x) = Θ(g(x)) if and only if
f (x) = O(g(x)) and f (x) = Ω(g(x)).
Moreover observe that Θ ⇒ O and Θ ⇒ Ω but the converses are false.
Example 2.5. We show: 3x lg x + 17 = O(x2 )
Consider the expression:

3x lg x + 17

Note two things:

• If x > 0 then lg x < x.
√
• If x ≥ 17 = 4.1231... then x2 ≥ 17.
Thus if x ≥ 5 both of these are true and we have:

3x lg x + 17 ≤ 3x(x) + x2 = 4x2

Thus x0 = 5 and C = 4 works.

Note: It’s not necessary to pick√an integer value of x0 here. I just did it
because it’s pretty. Using x0 = 17 would have been fine too.
100
Example 2.6. We show: x2 + x2 lg x = O(x3 )
Consider the expression:

100
+ x2 lg x
x2
Note two things:
• If x > 0 then lg x < x.
• If x ≥ 10 then x2 ≥ 100 and then 100
x2 ≤ 1 < x < x3 .
Thus if x ≥ 10 both of these are true and we have:

100
+ x2 lg x = O(x3 ) ≤ x3 + x3 = 2x3
x2
Thus x0 = 10 and C = 2 works.
Example 2.7. We show: 0.001x lg x + 0.0001x − 42 = Ω(x)
Consider the expression:

5
 0.001x lg x − 42

Note that if x ≥ 2 then lg x ≥ 1 and then:

0.001x lg x − 42 ≥ 0.001x − 42

This is a line with slope 0.001 and any line with smaller slope will eventually
be below it. For example the line 0.0001x is below it when:

0.001x − 42 ≥ 0.0001x
0.0009x ≥ 42
42
x≥ = 46666.66...
0.0009

Thus if we have x ≥ 46666.66... then:

0.001x lg x − 42 ≥ 0.001x − 42 ≥ 0.0001x

Thus x0 = 46667 and B = 0.0001 works.

Example 2.8. We show: 10x lg x + x2 = Θ(x2 )
Consider the expression:

10x lg x + x2

Observe that for all x ≥ 1 we have lg x > 0 and hence:

10x lg x + x2 ≥ x2

And we have:

10x lg x + x2 ≤ 10x(x) + x2 = 11x2

thus x0 = 1, B = 1 and C = 11 works.

For simple polynomials there’s very little work to show Θ.

Example 2.9. Observe that 3x2 = Θ(x2 ) because x0 = 0 and B = C = 3

works.
Example 2.10. Consider f (x) = 2x2 − x. Note that 2x2 − x ≤ 2x2 and

6
 2x2 − x ≥ 2x2 − x2 = 1x2 for x ≥ 1 so that x0 = 1, B = 1, C = 2 works for
2x2 − x = Θ(x2 ).

Example 2.11. Consider f (x) = 0.001x2 (1 + cos(xπ)).

The graph of this function is:

14

12

10

20 40 60 80

The local maxima occur at x = 0, 2, 4, 6, 8, ... and the local minima occur at
x = 1, 3, 5, 7, 9, ....
Note that 0.001x2 (1 + cos(xπ)) ≤ 0.001x2 (1 + 1) = 0.002x2 for x ≥ 0 so
that f (x) = O(x2 ). However in addition note that when x ∈ Z is odd that
0.001x2 (1 + cos(xπ)) = 0.001x2 (1 − 1) = 0 so that there is no B > 0 such
that for large enough x we have f (x) ≥ Bx2 . Consequently f (x) 6= Ω(x2 )
and thus f (x) 6= Θ(x2 ).
You might ask if there is any g(x) such that f (x) = Θ(g(x)) and the short answer
is - yes, of course, because f (x) = Θ(f (x)) but this is generally unsatisfactory.
We are looking for useful g(x) which help us understand f (x). Saying essentially
that f (x) grows at the same rate as itself doesn’t help much!

3 A Limit Theorem
There are a few alternative ways of proving O, Ω and Θ. Here is one. Note that
the following are unidirectional implications!
Theorem 3.0.1. Provided lim f (n) and lim g(n) exist (they may be ∞) then
n→∞ n→∞
we have the following:
f (n)
(a) If lim 6= 0, ∞ then f (n) = Θ(g(n)).
n→∞ g(n)
Note: Here we also have f (n) = O(g(n)) and f (n) = Ω(g(n)) as well.
f (n)
(b) If lim 6= ∞ then f (n) = O(g(n)).
n→∞ g(n)

7
 f (n)
(c) If lim 6= 0 then f (n) = Ω(g(n)).
n→∞ g(n)

Proof. Here’s a proof of (b). Suppose we have:

f (n)
lim = L 6= ∞
n→∞ g(n)

By the definition of the limit this means:

f (n)
∀ > 0, ∃n0 st n ≥ n0 =⇒ L −  < <L+
g(n)

Specifically, if  = 1 if we take only the right inequality this tell us that:

f (n)
∃n0 st n ≥ n0 =⇒ <L+1
g(n)

When < is true, so is ≤ so this means that when n ≥ n0 we have:

f (n) < (L + 1)g(n)

This is exactly the definition of O using n0 and C = L + 1. QED

Example 3.1. Observe that:

n ln n ln n 1/n
lim 2
= lim = lim =0
n→∞ n n→∞ n n→∞ 1
Thus n ln n = O(n2 ).
The following example is far easier to prove using this theorem than from the
definition of O:
Example 3.2. We have 50n100 = O(3n ).
Observe that 100 applications of L’hôpital’s Rule yields:

50n100 (100)(99)...(1)(50)
lim n
= lim =0
n→∞ 3 n→∞ (ln 3)100 3n

The result follows.

8
4 Common Functions
In all of this you might wonder why we’re always comparing functions to things
like n2 or n lg n. We typically wouldn’t say, for example, that f (n) = Θ(n2 +
3n + 1).

The reason for this is that computer scientists have settled on a collection of
“simple” functions, functions which are easy to understand and compare, and
big-notation almost always uses these functions.

Here are a list of some of them, in order of increasing size:

1, lg n, n, n lg n, n2 , n2 lg n, n3 , ...

To say these are “increasing size” means, formally, that any of these is O of
anything to the right, for example n = O(n lg n) and n2 = O(n3 ) and so on.

There’s a pattern there, that nk = O(nk lg n) and nk lg n = O(nk+1 ), which is

easy to prove.

In addition we have, for every positive integers k and b ≥ 2:

nk = O(bn )

These can be proved with the Limit Theorem.

Lastly, all of the above are O(n!), which is about the biggest one we ever en-
counter in this class.

5 Intuition
It’s good to have some intuition here, and of course the following can be proved
rigorously on a case-by-case basis, and you should try.

In essence the “largest term” always wins in a Θ sense. So for example if we

have:
f (n) = n2 − n lg n + n + 1
The “largest term” is the n2 so that wins and we can say:

n2 − n lg n + n + 1 = Θ(n2 )

Likewise, for example:

n2 lg n + n lg n − 100 = Θ(n2 lg n)

9
6 Additional Facts
6.1 Use of n vs x
These statements about function of x are often phrased using the variable n
instead. Typically this is done when n can only take on positive integers.

In this case it can still be helpful to draw the functions as if n could be any real
number, otherwise we’re left drawing a bunch of dots. In some cases though,
like f (n) = n!, it’s not entirely clear how we would sketch this for n 6∈ Z.

Otherwise the calculations are basically identical, noting that the cutoff value
n0 must be a positive integer.

6.2 Cautious Comparisons

This notation brings a certain ordering to functions. Observe for example that
1000000 + n lg n = O(n2 ) because eventually 1000000 + n lg n ≤ Cn2 for some
C > 0. Thus we intuitively think of n2 as “larger than” 1000000 + n lg n. How-
ever we have to make sure we understand that we really mean that a constant
multiple of n2 is eventually larger than 1000000 + n lg n.

Example 6.1. For example eventually 1000000 + n lg n ≤ 17n2 but eventu-

ally here means for n ≥ n0 = 243.
We should also note that it’s common to believe that if one function g(x) has a
larger derivative than another function f (x) that eventually f (x) ≤ g(x). This
is false.

10
7 Thoughts, Problems, Ideas
1. It’s tempting to think that if f (x) and g(x) are both positive functions
defined on [0, ∞) with positive derivatives and if f 0 (x) > g 0 (x) for all x
then eventually f (x) will be above g(x). Show that this isn’t true. Give
explicit functions and sketches of those functions.
2. Find the value x0 (approximately) which justifies 1234 + 5678x lg x =
O(x2 ) with C = 42. Use any technology you like but explain your process.
3. Find the value x0 (approximately) which justifies 4758x + 789x2 lg x =
O(x3 ) with C = 17. Use any technology you like but explain your process.
4. Find the value x0 (approximately) which justifies 0.00357x2.01 lg x = Ω(x2 )
with C = 100. Use any technology you like but explain your process.
5. Show from the definition that 5x2 + 10x lg x + lg x = O(x2 ).
6. Show from the definition that:
 
n−1
X n−1
X
2 + 3 = O(n2 )
i=0 j=i

7. Show from the definition that (475632)2n = O(5n ).

8. Show from the definition that x + x log x = Θ(x lg x).
9. Show from the definition that:
n−1
X 
1
1+i+ = Θ(n2 )
i=0
i+1

10. Show from the definition that x3 + 5x + ln x + 100 = Ω(x2 ).

11. Show from the definition that:
n
X 2
i + 3i = Ω(n3 )

i=0

12. Show from the definition that 5n 6= O(2n ).

13. Show that 5000 + 6000n1500 = O(3n ).
14. Show that 5n = Ω(n1000 )
15. Show that (0.001)5n = Ω(857n999 )
16. Show from the definition that log2 n = Θ(log5 n) and log5 n = Θ(log2 n).
17. Generalize the above problem. In other words prove that Θ(logb x) =
Θ(logc x) for any two bases b, c > 1.
18. In the previous question why do we need b, c > 1?

11
19. Give an example of two functions f (x) and g(x) which are not constant
multiples of one another and which satisfy f (x) = O(g(x)) and g(x) =
O(f (x)). Justify from the definitions.
20. Give an example of two functions f (x) and g(x) which are not constant
multiples of one another and which satisfy f (x) = Ω(g(x)) and g(x) =
Ω(f (x)). Justify from the definitions.
21. If f (n) = O(g(n)) with C0 and n0 and g(n) = O(h(n)) with C1 and n1
which constants would prove that f (n) = O(h(n))?
22. The functions f (x) = logb x for b > 1 and g(x) = xc for 0 < c < 1 have
similar shapes for increasing x. however f (x) = O(g(x)) always. Prove
this.
Note: This underlies the important fact that roots always grow faster than
logarithms.
23. Consider the following three functions:
f (x)

h(x)

g(x)

(a) Write down as many possibilities as you can which satisfy  = O(♦)
where , ♦ ∈ {f (x), g(x), h(x)}.
(b) Write down as many possibilities as you can which satisfy  = Ω(♦)
where , ♦ ∈ {f (x), g(x), h(x)}.
(c) Write down as many possibilities as you can which satisfy  = Θ(♦)
where , ♦ ∈ {f (x), g(x), h(x)}.

12