Theory of Computation

The Diagonalization Method

Teodor Rus
[email protected]

The University of Iowa, Department of Computer Science

a
Copyright Teodor Rus. These lecture notes have been developed by Teodor Rus using the book: Michael Sipser, Introduction to the Theory of Computation (second

edition), published by Thomson Course Technology 2006. They are copyrighted materials and may not be used in other course settings outside of the University of Iowa in their current form or modied form without the express written permission of one of the copyright holders. During this course, students are prohibited from selling notes to or being paid for taking notes by any person or commercial rm without the express written permission of one of the copyright holders.

Computation Theory. Copyrights Teodor Rus p.1/54

Decidability of TM Language
Problem:

For M a Turing machine and w a string, does M accept w?

Language:

AT M = { M, w |M is a TM that accepts w}

Computation Theory. Copyrights Teodor Rus p.2/54

Solution
Theorem 4.11:

AT M is recognizable but not

decidable.
A recognizer of AT M is the following TM called the Turing Universal Machine U : U = On input M, w , where M is a TM and w is a string: 1. Simulates M on w; 2. If M ever enters its accept state, accept; if M ever enters its reject state, reject ".
Note:

U is a universal TM because it simulates any other TM

from its description.

Computation Theory. Copyrights Teodor Rus p.3/54

Facts
1. So far we have tackled only solvable (decidable) problems; 2. Theorem 4.11 states that AT M is unsolvable (undecidable); 3. Since AT M is undecidable, to solve this problem we need to expand our problem solving methodology by a new method for proving undecidability.

Computation Theory. Copyrights Teodor Rus p.4/54

Diagonalization

The proof of undecidability of the halting problem uses Georg Cantor (1873) technique called diagonalization; Cantors problem was to measure the size of innite sets; The size of nite sets is measured by counting the number of their elements;

can we measure the size of an innite set by counting the number of its elements?
Question:

Computation Theory. Copyrights Teodor Rus p.5/54

Fact
The size of innite sets cannot be measured by counting their elements because this procedure does not halt!

Computation Theory. Copyrights Teodor Rus p.6/54

Example Innite Sets

The set of strings over {0, 1} is an innite set; The set N of natural number is an innite set; Both of them are larger than any nite set. How can we compare them? Design a mapping f : N {0, 1} by:
1. f (0) = ; 2. f (n) = {x {0, 1} ||x| = n}.

Computation Theory. Copyrights Teodor Rus p.7/54

Fact
f actually maps N on P({0, 1} ).
Since |x| = |y| for any x, y f (n), | | is an equivalence relation on {0, 1} and we can consider f : N {0, 1} /| |.

Computation Theory. Copyrights Teodor Rus p.8/54

Cantors Solution

Two nite sets have the same size if their elements can be paired; Since this method do not rely on counting elements it can be used for both nite and innite sets.

Computation Theory. Copyrights Teodor Rus p.9/54

The Correspondence
Consider two sets, A and B and f : A B a function.
f is one-to-one if it never maps two different elements of A into the same element of B, i.e., a, b A(a = b f (a) = f (b)); f is onto if it hits every element of B, i.e., b B, a A such that f (a) = b; f is called a correspondence if it is both one-to-one and onto.

a correspondence is a mechanism that allows us to pair elements of two sets.

Note:

Computation Theory. Copyrights Teodor Rus p.10/54

Set Size Comparison

Two sets A and B have the same size if there is a correspondence F : A B.

Computation Theory. Copyrights Teodor Rus p.11/54

Example Correspondences
Let N be the set of natural numbers, N = {1, 2, 3, . . .} and let E be the set of even natural numbers, E = {2, 4, 6, . . .}; Intuitively one may believe that size(N ) > size(E). However, using Cantor method we can show that N and E have the same size by constructing a correspondence f : N E; The correspondence f is dened by f (n) = 2n, Figure 1.

Computation Theory. Copyrights Teodor Rus p.12/54

sizeof (N ) = szeof (E)

n 1 2 3 ... f (n) 2 4 6 ...

Figure 1:
Note:

sizeof (N ) = sizeof (E)

1. n1 , n2 N , n1 = n2 , f (n1 ) = 2n1 , f (n2 ) = 2n2 i.e., f (n1 ) = f (n2 ); 2. n E, n = 2m, m N , i.e., n = f (m). Hence, f is a correspondence and thus sizeof (N ) = sizeof (E).

Computation Theory. Copyrights Teodor Rus p.13/54

Denition 4.12
A set is countable if either it is nite or it has the same size as N .

Computation Theory. Copyrights Teodor Rus p.14/54

A Complex Correspondence
Let Q be the set of positive rational numbers, Q = { m |m, n N }. n

Intuitively, Q seems to be much larger than N ; Yet we can show that sizeof (Q) = sizeof (N ) by constructing the correspondence c : N Q shown in Figure 2.

Computation Theory. Copyrights Teodor Rus p.15/54

Correspondence c : Q N
1. Put N on two axes; 2. Line i contains all rational numbers that have numerator i, i.e. { ji Q|i N f ixed, j N }; 3. Column j contains all rational numbers that have denominator j, i.e. { ji Q|i N , j N f ixed}; 4. Number
i j

occurs in i-th line and j-th column.

Computation Theory. Copyrights Teodor Rus p.16/54

Turning { |i, j N } into a List

i j

Bad idea: list rst elements of a line or a column.

Lines and columns are labeled by N , hence this would never end!

Good idea (Cantors idea): use the diagonals:

1 1 1. First diagonal contains 1 , i.e, rst element of the list is 1 ; 2 1 2. Continue the list with the elements of the next diagonal: 1 , 2 ;

3. Continue this way skipping the elements that may generate i repetitions, such i that would generate a copy of 1 . 1

Computation Theory. Copyrights Teodor Rus p.17/54

The List of Rational Numbers

               2 2 2 2 ... 2 1 2 3 4 5    2          3 3 3 3 3 ... 1 2 3 4 5   3           4 4 4 4 4 ... 1 2 3 4 5   4          5 5 5 5 5 ... 1 2 3 4 5   5   . .  .  . . . . ... .. ... .. ... . c
1

     E
1
1 1

2
1 2

3
1 3

4
1 4

5
1 5

...

...

Figure 2:

A correspondence of N and Q

Computation Theory. Copyrights Teodor Rus p.18/54

Uncountable Sets
A set for which no correspondence with N can be established is called uncountable.
Example of uncountable set:

The set R of real numbers is uncountable! Cantor proved that R is uncountable using the diagonalization method.
Proof:

Computation Theory. Copyrights Teodor Rus p.19/54

Example Uncountable Set

Theorem 4.17; R is uncountable. by contradiction using diagonalization. We will show that no correspondence exist between N and R.
Proof:

Suppose that such a correspondence f : N R exits and deduce a contradiction showing that f fail to work properly. We construct an x R that cannot be the image of any n N .

Computation Theory. Copyrights Teodor Rus p.20/54

Construction

Since f : N R is a correspondence R can be listed as seen in Figure 3

n 1 2 3 4 ... f (n) 3.14159 . . . 55.5555 . . . 0.1234 . . . 0.5000 . . . ...

Figure 3:

Listing R

Notation:

for x R, di (x) is the i-th digit of x.

Computation Theory. Copyrights Teodor Rus p.21/54

Formal Construction of x
Construct x (0, 1) by the following procedure:

x = 0.d1 d2 d3 d4 . . . where i N [di (x) = di (f (i))] x has an innite number of decimals constructed by the rule:
Note:
i N chose di a digit different from the i-th digit of f(i)

Consequence: i N , x = f (i). Hence, x does not belong to the list R; thus f is not a correspondence.

Computation Theory. Copyrights Teodor Rus p.22/54

Application
Theorem 4.17 shows that some languages are not decidable or even Turing recognizable.
Reason:
There are uncountable many languages yet only countable many Turing machines. Because each Turing machine can recognize a single language and there are more languages than Turing machines some languages are not recognized by any Turing Machine. Such languages are not Turing recognizable!

Computation Theory. Copyrights Teodor Rus p.23/54

Corollary 4.18
Some languages are not Turing-recognizable.
Proof:

First we show that the set of Turing machines is countable

1. The set of all strings is countable, for any alphabet . we may form a list by writing down all strings of length 0, length 1, length 2, an so on.
Proof:

2. Each Turing machine M has an encoding into a string M ; 3. If we omit those strings that are not Turing machines we can obtain a list of all Turing machines.

Computation Theory. Copyrights Teodor Rus p.24/54

Fact 1
The set of all languages is uncountable. To show that the set of all languages is uncountable we show rst that the set of all innite binary sequences is uncountable.
Proof idea:

Computation Theory. Copyrights Teodor Rus p.25/54

Innite Binary Sequences

Let B be the set of all innite binary sequences.

Assuming that B is countable we can set it into a list fb : N B. By the method of diagonalization we can construct an innite binary string y, such that y = fb (i) for any i N ;
Construction:

y = d1 d2 . . . dj . . . such that

ij[dj {0, 1} dj = dj (fb (i)]

Computation Theory. Copyrights Teodor Rus p.26/54

Corollary
The set of all languages is uncountable. by construction: Let L be the set of all languages over .
Proof:

We will show that L is uncountable by constructing a correspondence B L. Since B is uncountable, and has the same size as L it result that L is uncountable.

Computation Theory. Copyrights Teodor Rus p.27/54

Characteristic Sequences
Since is an alphabet, is countable. Let = {s1 , s2 , s3 , . . .} be the set of all strings over ; For each language A L there is a unique innite binary sequence A B constructed by: the i-th bit of A , A (i) = 1 if si A and A (i) = 0 if si A. A is the characteristic function of A in The function f : L B where f (A) = A is one-to-one and onto and hence it is a correspondence.
Proof:

1. f is one-to-one: L1 , L2 L, L1 = L2 L1 = L2 ; 2. f is onto: B there is a language L L with f (L) = . For = {s1 , s2 , . . .}, L = {si |si di () = 1}.

Computation Theory. Copyrights Teodor Rus p.28/54

Conclusion
Since B is uncountable, L is uncountable.

Computation Theory. Copyrights Teodor Rus p.29/54

Example characteristic function

Let = {0, 1} and A be the language of all strings starting with 0 over .
\Sigma*={e,0,1,00,01,11,000,001,010,011,100,...}; A \chi_A ={ 0, ,00,01, 1 0 ,000,001,010,011, 1 1 1 1 , ...}; ...

={0 1 0 1

Computation Theory. Copyrights Teodor Rus p.30/54

Fact 2
For M a TM, the language L(M ) is undecidable. We are ready to prove that the language
AT M = { M, w |M is a T M and M accepts w}

is undecidable.

Computation Theory. Copyrights Teodor Rus p.31/54

Proof
Proceeds by contradiction, assuming that AT M is decidable.

Suppose that H is a decider of AT M . On input M, w where M is a TM and w is a string, H halts and accepts if M accepts w. Furthermore, H halts and reject if M fails to accept w.

Computation Theory. Copyrights Teodor Rus p.32/54

Equational Expression of H
accept, if M accepts w; H( M, w ) = reject, if M does not accept w.

Computation Theory. Copyrights Teodor Rus p.33/54

Proof, Continuation
Construct a new TM D that uses H as a subroutine.

D calls H to determine what M does when its input is M If M accepts M then D rejects; if M rejects M then D accepts.

Computation Theory. Copyrights Teodor Rus p.34/54

The Machine D
D = "On input M , where M is a TM: 1. Run H on input M, M 2. Output the opposite of what H outputs: accepts if H rejects and rejects if H accepts".

Computation Theory. Copyrights Teodor Rus p.35/54

Note

Running a machine on its own description is a common technique in computer sciences. Example, running a compiler on its own description allows compiler implementation and optimization.

Computation Theory. Copyrights Teodor Rus p.36/54

In Conclusion
accept, if M does not accept M ; D( M ) = reject, if M accepts M .

What happens when we ran D on D ?

accept, if D does not accept D ; D( D ) = reject, if D does accept D .

This is a contradiction and consequently neither TM D nor TM H do exist.

Computation Theory. Copyrights Teodor Rus p.37/54

Summarizing

Assume that H decides AT M ; Use H to build D that accepts M when M rejects it and rejects M when M accepts it; H and D performs as follows:
H accepts M, w exactly when M accepts w; D rejects M exactly when M accepts M ; D rejects D exactly when D accepts D .

This is a contradiction and neither H nor D can exist.

Computation Theory. Copyrights Teodor Rus p.38/54

Where Is Diagonalization?
To make the use of diagonalization obvious we construct the list of all Turing machines running on Turing machines as input in Figures 4,5,6.
M1 M2 M3 M4 M1 accept accept accept M2 accept accept M3 accept accept M4 accept ... ... ...

Figure 4: iEntry (i,j) is accept if Mi accepts

Mj

Computation Theory. Copyrights Teodor Rus p.39/54

Running H
Figure 5 shows the result of running H on the machine in Figure 4.
M1 M2 M3 M4 M1 accept accept reject accept M2 reject accept reject accept M3 accept accept reject reject M4 reject accept reject reject ... ... ... ... ...

Figure 5:

Entry (i,j) is the value of H on Mi , Mj

Computation Theory. Copyrights Teodor Rus p.40/54

Running D on D
Figure 6 shows the result of running H on the machine in Figure 4 when D is present.
M1 M2 M3 M4 D M1 accept accept reject accept reject M2 reject accept reject accept reject M3 accept accept reject reject accept M4 reject accept reject reject accept ... ... ... ... ... ... D accept accept reject accept ??? ... ... ... ... ... ...

Figure 6:

A contradiction occurs at D, D

Computation Theory. Copyrights Teodor Rus p.41/54

Observation
We can construct a Turing-unrecognizable language.
AT M is an example of Turing undecidable language. But it is Turing recognizable; Now we construct a language which is Turing-unrecognizable; This construction relies on the fact that if both a language and its complement are Turing-recognizable the language is decidable.

That is:

for any undecidable language, either the language or its complement is not Turing-recognizable.

Computation Theory. Copyrights Teodor Rus p.42/54

A New Concept
Co-Turing-recognizable language.

Complement of a language A is the language consisting of all strings that does not belong to A; A language is co-Turing-recognizable if it is the complement of a Turing-recognizable language.

Computation Theory. Copyrights Teodor Rus p.43/54

An Important Result
A language is decidable iff it is both Turing-recognizable and co-Turing recognizable.
Theorem 4.22:

That is: A language A is decidable iff both A and C(A) are Turing-recognizable

Computation Theory. Copyrights Teodor Rus p.44/54

Proof
if:
Assume that A is decidable. Since complement of a decidable language is decidable it result that both A and C(A) are Turing-recognizable. For L decidable, decided by M , the TM M that decides L is: M = On any input w: 1. Run M on w 2. If M accepts reject; if M rejects accept".

Computation Theory. Copyrights Teodor Rus p.45/54

Proof, Continuation
only if:
Assume that both A and C(A) are Turing-recognizable. Let M1 be a recognizer for A and M2 a recognizer for C(A). Then the following TM M is a decider for A: M = "On input w: 1. Run both M1 and M2 on w in parallel; 2. If M1 accepts w accept; if M2 accepts w reject."

Computation Theory. Copyrights Teodor Rus p.46/54

Observations
Running two machines M1 and M2 by a machine M in parallel means that M has two tapes, one for simulating M1 and other for simulating M2 ; M takes turns, simulating one step of each machine, which continues until one of the machines halts; Because w A or w C(A) either M1 or M2 must accepts w; Because M halts whenever M1 or M2 accepts, M always halts, so it is a decider. Further, it accepts all strings from A and rejects all strings not in A.

Computation Theory. Copyrights Teodor Rus p.47/54

Conclusion
M is a decider for A, thus A is decidable.

Computation Theory. Copyrights Teodor Rus p.48/54

Corollary
C(AT M ) is not Turing-recognizable.
Proof:

We know that AT M is Turing-recognizable.

1. If C(AT M ) also were Turing-recognizable then AT M would be decidable. 2. But we have proved (Theorem 4.11) that AT M is not decidable. 3. Hence, C(AT M ) cannot be Turing-recognizable.

Computation Theory. Copyrights Teodor Rus p.49/54

Application 1
Let A be a Turing-recognizable language consisting of descriptions of Turing machines:
A = { M1 , M2 , . . .}

where every Mi is a decider. Prove that some decidable language LD is not decided by any decider Mi whose description appears in A.

Computation Theory. Copyrights Teodor Rus p.50/54

Solution Sketch
Use the diagonalization method to construct a decider D whose language is not among those decided by Mi , i = 1, 2, . . .. Since A is Turing-recognizable there is an enumerator E that enumerates the elements of A. Let M1 , M2 , . . . be the output of E. Let = {s1 , s2 , . . .} be a list of all strings over the alphabets of A.
The decider for D is: D = On input w: 1. Let i be the index of w on the list of strings , i.e., si = w. 2. Run Mi on input w 3. If Mi accepts, reject; if Mi rejects, accept".
Note:

D is a decider because each Mi is a decider. But D does not appear on the list

enumerated for A, because i, D differs from Mi on input si

Computation Theory. Copyrights Teodor Rus p.51/54

Application 2
Let A and B be two disjoint languages. Say that a language C separates A and B if A C and B C. Show that any two disjoint co-Turingrecognizable languages are separable by some decidable language.

Computation Theory. Copyrights Teodor Rus p.52/54

Solution Sketch
Let A and B be two languages such that A B = and A and B are Turing-recognizable (by denition). Let J be the TM that recognizes A and K be the TM that recognizes B. Then the language C decided by the following TM T separates A and B:
T = On input w: 1. Simulate J and K on w by alternating the steps of the two machines; 2. If J accepts rst, reject. If K accepts rst, accept".

Computation Theory. Copyrights Teodor Rus p.53/54

L(T ) Separates A and B

1. A = \ A, B = \ B, A B = \ A B. Since A B = it results that A B = . 2. A B = implies that T is a decider because for any w , w A or w B. Hence, either J or K will accept w. 3. A L(T ) because if w A, w will not be recognized by J and will be accepted by K rst, i.e., w L(T ). 4. B L(T ) because if w B, w will not be recognized by K and will bed accepted by J rst, i.e., w L(T ).
Conclusion:

by the denition of separability, language C = L(T ) sepa-

rates A and B

Computation Theory. Copyrights Teodor Rus p.54/54