Introduction

CS240 • You are about to embark on the study of a

fascinating and important subject: the theory
of computation
Language Theory – Fundamental mathematical properties of computer hardware,
software, and certain applications of these

And Automata – Seek to determine

• What can and cannot be computed
• How quickly something can be computed
• How much memory is needed to compute something
• Which type of computational model can be used
Fall 2006
Obvious connections with engineering practice, but also purely
philosophical aspects
2

Why study this stuff? Theory is relevant to practice

• Theoretical computer science has • Provides conceptual tools that practitioners
many fascinating big ideas, but also use in computer engineering
– Design a new programming language for a special
many small and sometimes dull application - need grammars!
details – String searching and pattern matching - use finite
– The more you learn, the more interesting it automata and regular expressions!
becomes – Deal with a problem that seems to require more
– Our goal: be exposed to the exciting aspects of computer time than you can afford - consider NP-
computer theory without getting bogged down in completeness!
drudgery

3 4

1
 Theory shows the more elegant
Theory expands your mind
side of computers
• Usually think of computers as complicated • Computer technology changes quickly
machines • Specific technical knowledge becomes
• Best computer designs and applications are outdated in a few years
conceived with elegance in mind • Studying theory trains you in the abilities to
think, express yourself clearly and precisely,
• A theoretical course can heighten your solve problems, know when you have not
aesthetic sense and help you build more solved a problem
beautiful systems – These abilities have lasting value

5 6

Automata, Computability, and

Automata Theory
Complexity
• Linked by the question: What are the • We start with automata theory
fundamental capabilities and limitations of
– Theories of computability and complexity
computers?
require a precise definition of a computer
• Each area interprets the question differently
– Automata theory allows practice with formal
– Complexity: is a given problem easy or hard?
– Computability: is a given problem solvable or unsolvable? definitions of computation as it introduces
• E.g., no computer can solve the problem of whether a concepts relevant to other non-theoretical
mathematical statement is true or false
areas of computer science
– Automata theory: definitions and properties of mathematical
models of computation

7 8

2
 Course Overview Mathematical Background
• Study categories of languages and • Set theory
machines • Relations and functions
– Regular languages and Finite automata
• Recursive definitions
– Context-free languages and Push-down automata
– Unrestricted languages and Turing machines • Strings and languages
• Study solvability and efficiency • Proofs
– The Halting problem
– NP-completeness

9 10

Sets
Sets
Be able to determine the union (∪), intersection (∩),
difference (-), complement (overbar), and power set
• Group of objects represented as a unit
|P(X)| = 2X of any given sets.
• May contain any type of object: numbers,
symbols, other sets,… Proving equality of sets X and Y: show every
– Set membership: ∈ element of X is an element of Y and vice versa (i.e.,
– Non-membership: ∉
show that X ⊆ Y and Y ⊆ X to show X = Y.)
– Subset: ⊆
• Proper subset
– Empty set: ∅ If X is a set, the notations |X| and card(X) both
– Infinite set contains infinitely many elements refer to the number of elements in X.
• E.g., set of integers {…-2, -1, 0, 1, 2,…}

11 12

3
Cartesian Product and Relations Functions
A function from a set X to a set Y is a mapping of
elements of X to elements of Y such that each
Cartesian product builds a set consisting of ordered element of X maps to exactly one element of Y.
pairs of elements from two or more existing sets. f:X→Y
X × Y = {[x, y] | x ∈ X and y ∈ Y}...(a set of pairs) X is the domain of f.
The range of f : X → Y is the set {y ∈ Y | y = f (x) for
A binary relation on sets X and Y is a subset of some x ∈ X}.
X×Y.
A total function f from X to Y is a binary relation on
X × Y such that
An n-ary relation on sets X1, X2, ..., Xn is a subset of i. For each x ∈ X there is a y ∈ Y such that [x, y] ∈ f.
X1 × X2 ×... × Xn ii. If [x, y] ∈ f and [x, z] ∈ f, then y = z.

13 14

Total Functions Countable and Uncountable Sets

f : X → Y is one-to-one if each element of We divide sets into classes:

X maps to a distinct element in the • Countably infinite (denumerable): has same
range. number of elements as N.
• Countable: finite or countably infinite.
f : X → Y is onto if the range is the entire
• Uncountable: has more elements than N.
set Y.
To show a set A is countably infinite, we need to show a
one-to-one function exists from N to A.

15 16

4
 Uncountable Sets Recursive Definitions
Prove a set X is uncountable by showing that it Recursion provides a method for generating elements
is impossible to sequentially list its members of a set by specifying:
Cantor’s diagonalization argument: used to • basis elements explicitly
show that a set is uncountable. Proof by
contradiction. • a finite set of operators (aka functions), used to
construct the remaining elements of the set from
Step 1: Assume the set is countable and therefore its members can be
exhaustively listed. This listing must contain all members of the the basis elements.
set.
Step 2: Produce a member of the set that cannot occur anywhere in the Generation using a finite but unbounded number of operations
listing, showing that such a listing cannot exist and therefore the is a fundamental property of recursive definitions.
set is uncountable.
17 18

Recursive Definitions Recursive Definitions

Example: Example:
A recursive definition of N, the set of natural A recursive definition of the sum of m and n
numbers using the successor function, defined using the successor function:
as: s(n) = n+1: 1. Basis: If n = 0, then m + n = m.
1. Basis: 0 ∈ N. 2. Recursive step: m + s(n) = s(m + n).
2. Recursive step: If n ∈ N, then s(n) ∈ N 3. m + n = k only if the equality can be obtained from
m + 0 = m by a finite number of applications of the
3. n ∈ N only if it can be obtained from 0 by a finite
recursive step.
number of applications of the recursive step.
In step 2, the sum of m and the successor of n is defined in
Elements are: 0, s(0), s(s(0)), s(s(s(0))), ... terms of the simpler case, the sum of m and n.
0, 1, 2, 3, ... 19 20

5
 Languages Examples of Languages
• Alphabet (Σ) : finite set of symbols, e.g., • The set of all binary strings consisting of
- Σ = {0, 1} (binary alphabet) some number of 0's followed by an
- ASCII equal number of 1's; that is, ε; 01; 0011;
• String : finite sequence of symbols 000111; …
chosen from some alphabet, e.g., 01101
or abracadabra. • C (the set of C programs that compiles
without syntax errors).
• Language : set of strings chosen from
some alphabet • English.

21 22

Strings
• Empty string : ε (may be λ in some texts)
• Length : • Concatenation : Given strings x and y,
• |abcde|= 5 concatenation is xy
• |uv| = |u| + |v|
– E.g., x = abc, y = def, xy = abcdef
• Reverse : wR
• If w= abc, wR = cba
• Powers
• Substring : any string of consecutive characters − Σk = set of all strings from alphabet Σ with length k
in some w − Σ0 = {ε}
• Prefix and suffix : if w = vu, v and u are a prefix − Σ* = set of all strings from alphabet Σ
and a suffix of w, respectively − Σ+ = Σ * - {ε}
Convention: Use lower case letters from the beginning of the alphabet for
symbols, lower case letters from end of alphabet for strings
23 24

6
 More About Languages
Proofs
• Concatenation of languages
– L1L2 = {xy | x ∈ L 1, y ∈ L2} • The only way to prove the truth or falsity
– Ln = L concatenated with itself n times of a mathematical statement is with a
• L0 = {ε}; L1 = L
proof
• Star-closure
– L* = L0 ∪ L1 ∪ L2 ∪ … (note: L+ = L * - L0) • Finding proofs isn’t always easy!
• Languages can be finite or infinite • Helpful strategies:
– L = {a, aba, bba} – Carefully read the statement you want to prove
– L = {an | n > 0} – Rewrite the statement in your own words
– Break the statement down and consider each part
Subtle point separately
A language may be infinite, but there is some finite set of symbols of
which all its strings are composed (so Σ is always a finite set) 25 26

Parts of proofs Proving the Equivalence of Sets

• Parts of a multi-part statement are not • Many important facts in language theory
always obvious are of the form that two sets of strings,
• Frequent multi-part statement: described in two different ways, are
– P if and only if Q (P iff Q ; P ⇔ Q) really the same set. To prove sets S and T
– Two-part statement: are the same, prove:
• If P is true, Q is true (P ⇒ Q) forward direction • x is in S iff x is in T. That is:
• If Q is true, P is true (P ⇐ Q) reverse direction • Assume x is in S; prove x is in T.
– To prove iff statement, must prove both directions • Assume x is in T; prove x is in S.

27 28

7
 Example More strategy for doing proofs
• Prove one of DeMorgan’s Laws • Get an intuitive “gut” feeling of why the
– For any two sets A and B, A ∪ B = A ∩ B statement should be true
• Is the meaning clear?
• Experiment with examples
• Recall: Can prove two sets equal by showing that every
– E.g., if statement says all objects of a certain type have a
member of one is a member of the other and vice versa particular property, pick a few objects of that type and
PROOF observe that they actually have that property
– Then, try to find an object that fails to have the property
Suppose that x is an element of A ∪ B. Then x is not in A∪B by definition of
complement. Therefore x is not in A and x is not in B, by the definition of (counterexample)
union. In other words, x is in A and x is in B. The definition of intersection of • If the statement actually is true, you won’t be able to find a
two sets shows that x is€in A ∩ B . counterexample
Other direction: suppose x is in A ∩ B . Then x is in both A and B . Therefore x is • Seeing where you run into trouble when you try to find a
not in A and x is not in B, and therefore
€ not in A∪B. Hence x is in the counterexample can help understand why the statement is true
€
complement of the € union of these sets; in other words, x is in A ∪ B which
completes the proof of the
€ theorem. € €
29 30
€

Example
Prove: For every graph G, the sum of the degrees of all the
Still stuck?
nodes is an even number • Attempt to prove a special case of the
• Step 1: Pick a few graphs and observe the statement in statement
action

Example
Sum = 2+2+2=6 Sum = 2+3+4+3+2=14
Trying to prove some property is true for
every k > 0
• Step 2: Try to find a counterexample – Try to prove for k = 1
Sum = 1+1=2 – If you succeed, try for k = 2, etc. until you
Sum always understand the more general case
Sum = 1+2+1=4 increases by 2!
Sum = 1+3+1+1=6
– If not, try another special case
31 32

8
 A Few Tips
Writing up the proof
• Be patient
• A well-written proof is a sequence of – Finding proofs takes time. Don’t worry if you don’t see how to
do it right away.
statements, wherein each one follows by
simple reasoning from previous statements. • Come back to it
– Let the unconscious, intuitive part of your brain have a chance
• Important: Explicitly provide reasons after each to work
algebraic manipulation. • Be neat
– Use simple, clear pictures and text. Neatness helps both you
• Carefully writing a proof : and the person who has to read your proof (me!) understand it
– Enables reader to understand
• Be concise
– Assures you the proof is free from errors
– Brevity helps express high-level ideas without getting lost in
details. Use good mathematical notation. But be sure to
include enough of your reasoning to make the argument clear.
33 34

Types of Proof Proof by Induction

• Advanced method to show that all elements of an
• Proof by construction infinite set have a specified property
– Theorem states that a particular type of object exists • Two parts:
– Prove by demonstrating how to construct such an object – Each part is an individual proof on its own
– Induction step : Prove that for each i ≥ 1, if P(i) is true, then so is
• Proof by contradiction P(i+1)
– Assume the theorem is false and show this leads to an – Basis: Proves that P(1) is true
obviously false consequence • When both parts are proven the desired result follows,
– Do this in everyday life: that P(i) is true for each i.
• Upon seeing someone who has just come in from outdoors and
– WHY DOES THIS WORK?
is completely dry, we know it is not raining. “Proof” is that if it
• We know that P(1) is true because the basis alone proves it.
were raining (the assumption that the statement is false), the
person who just came in would be wet (the obviously false • We know that P(2) is true because the induction step proves that, if P(1)
consequence). Therefore it must not be raining. is true then P(2) is true, and we already know P(1) is true.
• We know that P(3) is true because the induction step proves that, if P(2)
is true then P(3) is true, and we already know P(2) is true.
35 • and so on…. 36

9
 Example
• A binary tree with n leaves has 2n - 1 nodes
• Formally, S(T): if T is a binary tree with n leaves, then T has 2n -
1 nodes.

APPENDIX
• Induction is on the size = number of nodes of T .

APPENDIX
– Basis: If T has 1 leaf, it is a one-node tree. 2(1) –1 = 1.

APPENDIX
– Induction: Assume S(U) for trees with fewer nodes than T . In
particular, assume for the subtrees of T .
• T must be a root plus two subtrees U and V .
• If U and V have u and v leaves, respectively, and T has t leaves,
then u + v = t.
• By the inductive hypothesis, U and V have 2u - 1 and 2v - 1
nodes, respectively.
• Then T has 1 + (2u - 1)+ (2v - 1) nodes
• = 2(u + v) -1
• = 2t - 1, proving the inductive step. 37 38

Another Proof Example Theorem: A string of parentheses w is

Balanced Parentheses GB if and only if it is SB.
• Here are two ways that we can define
"balanced parentheses":
• If
– An induction on |w| (length of w). Assume w is SB;
1. Grammatically :
a) The empty string ε is balanced. Recursive prove it is GB.
definition • Basis: If w = ε (length = 0), then w is GB by rule (a).
b) If w is balanced, then (w) is balanced.
c) If w and x are balanced, then so is wx. – Notice that we do not even have to address the question of
2. By Scanning: w is balanced if and only if: whether ε is SB (it is, however).
(a) w has an equal number of left and right parentheses. • Induction: Suppose the statement “SB implies GB" is true
(b) Every prefix of w has at least as many left as right parentheses. for strings shorter than w.
Example prefixes: ε, (, (), ()(, ()()….
ε, (, ((, (((, (((), ((())…

Call these GB and SB properties, respectively.

39 40

10
 • Case 2:
• Case 1:
– w = xy, where x is the shortest non-empty prefix of w
– w is not ε, but has no non-empty prefix that has an with an equal number of ( and ), and y is not ε.
equal number of ( and ). Then w must begin with (
• x and y are both SB (why?).
and end with ); i.e., w = (x).
• x must be SB (why?).
• By the IH, x and y are GB.
• Because of the assumption that w is SB, I.e., has an equal number
of left and right parentheses and adheres to the prefix property.
Eliminating the outer left and right parentheses of w retains the SB • w is GB by rule 1(c).
properties.
• By the IH, x is GB.

• By rule 1(b), (x) is GB; but (x) = w, so w is GB.

41 42

• Only-If • Case 1:
– w is GB because of rule 1(b); i.e., w = (x) and x is
GB.
– An induction on |w|. Assume w is GB; prove it is SB.
• by the IH, x is SB.
• Since x has equal numbers of ('s and )'s, so does (x).
– Basis: w = ε . Clearly w obeys the conditions for being SB.
• Since x has no prefix with more ('s than )'s, so does (x).

– Induction: Assume “GB implies SB" for strings shorter than w,

and assume w is not ε.

43 44

11
• Case 2:
– w is not ε and is GB because of rule (c); i.e., w = xy,
and x and y are GB.
• By the IH, x and y are SB.
• (Aside) Trickier than it looks: we have to argue that neither x
nor y could be ε, because if one were, the other would be w,
and this rule application could not be the one that first shows
w to be GB.
• xy has equal numbers of ('s and )'s because x and y both do.
• If w had a prefix with more )'s than ('s, that prefix would
either be a prefix of x (contradicting the fact that x has no
such prefix) or it would be x followed by a prefix of y
(contradicting the fact that y also has no such prefix).

Above is an example of proof by contradiction. We assumed our

conclusion about w was false and showed it would imply something that
we know is false.
45

12