Unformatted text preview: Chapter 1
Introduction
Theory of computation is an attempt to formalize the notion of computation, i.e., the notion of an
algorithmic process. Computational Complexity aims to determine the solvability/unsolvability of
computational problems, to develop techniques for discovering e cient algorithms, and to explain
why some problems are inherently di cult in terms of the time and/or space they take to solve.
For example, the problem of sorting numbers by comparison, can be solved in O(n log n) time
by using merge sort. Since this bound has been achieved, it can be viewed as an upper bound the question that remains is  can we do better? It has been shown that if n numbers are sorted
using comparisons, then we need at least O(n log n) steps. Hence, for this problem the lower bound
has been established and realized, and merge sort is an asymptotically optimal comparisonbased
algorithm.
Consider the problem of multiplying two n n matrices. A naive algorithm requires O(n3 )
operations. Is this the best we can do? Other more sophisticated algorithms are known which take
less time: there is an algorithm to multiply two n n matrices using O(n2:376:::) operations. Can
we do better? The answer to this question is not known since no nontrivial lower bound has been
established for matrix multiplication. A trivial lower bound is O(n2 ).
The partition problem is de ned as follows. Given a set of positive integers, c1 c2
cn , is it
possible to partition them into two sets, P1 P2 , such that the sum of the integers in each set is the
same i.e.,
X
X
ci =
ci ?
ci 2P1 ci 2P2 Although this looks like a simple problem, all known algorithms require exponential running time.
This is an example of a what are called NPcomplete problems.
Finally let us consider Hilbert's Tenth Problem (HTP) and its application to the equivalence
problem for programs. De nition 1 Hilbert's Tenth Problem (HTP):
Given: A polynomial p(x1 x2 xn) of n variables and integer coe cients (e.g. p(x1 x2 x3 ) =
1 c 2 O.H.Ibarra 45x2 x2 ; 3x1 x3 + x2 ; 5).
1
2
Question: Does p(x1 x2
that p(a1
an ) = 0)? xn) have an integral root (i.e. are there integer values a1 an such Is HTM decidable? I.e. does there exist an algorithm which when given an arbitrary polynomial p
will answer Yes if p has an integral root and No otherwise?
Yes (p has an integral solution)
p
ALGO
No (p does not have an integral solution) If p is over a single variable then it is decidable. Why? The problem was posed in 1900 and
in 1970 it was shown to be undecidable, i.e., it was shown that no algorithm exists. It should also
be noted that the result holds for solutions over the set of nonnegative integers, f0 1 2 g. The
proof of this is left to the reader. Hint: To show that an algorithm (i.e. decidability) for nonnegative integers case implies an algorithm (decidability) for the integers case, look at all possible
combinations of positive and negative values for the variables. For the converse, you'll need the
triangular integer hint: Any nonnegative integer can be represented as the sum of 3 triangular
numbers. n is a triangular number if n = i(i+1) , for some integer i (+,, or 0).
2
Two applications of the undecidability of HTP are now presented.
1. Given: two nonlooping programs (i.e., programs without loops in some programming language) P1 and P2 .
Question: Are they equivalent, i.e., is P1 P2? This is the problem of Equivalence of
Programs.
We show that the equivalence of programs is undecidable (over the integral domain).
Proof: Suppose the equivalence of programs is decidable, i.e., 9 an algorithm for it. Then we
will show that there is an algorithm for HTP, which is a contradiction since HTP is known
to be undecidable.
Suppose p(x1
xn) is a polynomial. We de ne the following two programs: P1 : input(x1
xn)
y p(x1
xn ) /* use +,,* to compute the value of p and store in y */
if y 6= 0 then output(`0') end else output(`1') Hence P1 will always output `0' if p(x1 xn) has no solution. c 3 O.H.Ibarra P2 : input(x1 xn ) output(`0')
end Thus we see that P1 P2 if and only if p(x1 xn ) has no solution. Corollary 1 6 9 an algorithm (i.e. it is unsolvable) for program optimization.
Proof: Suppose 9 an algorithm for program optimization. Given a polynomial p(x1 xn), we construct the program P1 as described above. We give this program as input to the
optimization algorithm which gives as output the program P1optimal , the optimized version of
P1 . Now P1optimal will be the same as P2 if and only if p(x1
xn) has no solution. Hence
using the optimization algorithm, we can solve HTP  which is a contradiction. Hence no
algorithm for optimization exists.
2. The second application is in Number Theory. A logician named Alonzo Church showed that
there is no algorithm (i.e., it is undecidable) that can decide, in general, whether statements
in number theory are true or false. To be more precise: De nition 2 Let the universe (or domain) be the set of nonnegative integers, N = f0 1 2 3 g.
Statements (or formulas) are built using operations + , quanti ers 9 8, connectives _ ^ : over the universe N and variables x1 x2 . Examples:
8x9y x + x = y]  this is a True statement.
9y8x x + x = y]  this is a False statement.
8q9p8x y p > q ^ (x y {z 1 ) xy = p}]  this is a True statement (it states that there
>
6
can be rewritten using only_^ and : are an in nite number of primes). The formulas can be written such that all quanti ers appear in front of the formula such as: F = Q1 x1 Q2 x2 Qn xn ] where Qi is either 9 or 8, is a formula without quanti ers over variables x1 xn . Theory(N + )
is the set of all true formulas over N using only the operators + and . Theory(N +) is the
set of all true formulas over N using only the operator +.
Claim: Theory(N + ) is undecidable, i.e., 9 no algorithm to decide given a formula, F ,
in Theory(N + ) whether F is true or false. (Thus some theorems or true statements in
Theory(N + ) are not provable.) In other words, theorem proving cannot be \mechanized". c 4 O.H.Ibarra Proof: Given a polynomial p(x1 xn), let p1 (x1
xn) be the positive terms and p2(x1
be the negative terms. Then de ne the formula F to be:
F = 9x1 9x2 9xn p1(x1 xn ) = p2 (x1 xn)] E.g. For p = 45x2 x2 ; 3x1 x3 + x2 ; 5, p1 = 45x2 x2 + x2 and p2 = 3x1 x3 +5. The corresponding
1
2
1
2
formula would be:
9x19x29x3 45x2 x2 + x2 = 3x1 x3 + 5]
1
2
which can be simpli ed by introducing more variables. For example: 9x19x29x39z 45x2 x2 + x2 = z ^ z = 3x1x3 + 5]
1
2
9x19x29x39z9w 45x2 x2 + w = z ^ z = 3x1 x3 + 5 ^ w = x2 x2 ]
1
Now F is true if and only if p has a solution. Thus if there exists an algorithm to determine
whether F is true or false, then we can use it to determine whether p has a solution, which is a
contradiction. Hence there is no algorithm to decide the validity of formulas in Theory(N,+,*).
Claim: Theory(N +) is decidable.
Proof: (Sketch) Consider the formula F = 9x8y9w9zE , where E is the expression (x + x =
y) ^ (x + y = z ) _ :(z = w)] Suppose we want to check whether F is true. Denote by: E1 the subexpression x + x = y
E2 the subexpression x + y = z
E3 the subexpression z = w
De ne the following languages: L1 = f(x y z w)jx y z w in N x + x = yg
L2 = f(x y z w)jx y z w in N x + y = z g
L3 = f(x y z w)jx y z w in N z = wg
x y z w will be represented in binary with leading ( rst) bit the LSB. So, for example,
x = 5 is 1010
y = 7 is 1110
z = 3 is 1100
w = 12 is 0011 xn ) c 5 O.H.Ibarra Note that we can assume that the binary numbers have the same length. Thus, the tuple
(5 7 3 12) in binary can be represented as (1 1 1 0)(0 1 1 0)(1 1 0 1)(0 0 0 1). Thus the
binary representation of (5 7 3 12) is a string over the alphabet, 4 = f(s1 s2 s3 s4 )jsi 2
f0 1gg The \binary" representation (version) of L1 is the language over the alphabet 4:
f(a0 b0 c0 d0 )(a1 b1 c1 d1 ) (an bn cn dn )jx = a0 an y = b0 bn z = c0 cn w =
d0 dn are binary strings and x + x = yg.
For notational convenience we will also use L1 to denote its binary representation.
We can construct a DFA A1 to accept L1 . A1 simply reads the input string and performs the
binary addition (in this case adding x to itself) and checking that the binary sum is equal
to y. A1 only looks at the rst and second tracks and ignores the third and fourth tracks.
Similarly, we can construct DFA's A2 and A3 to accept L2 and L3 , respectively.
Now let E4 be the expression :E3 , then the language L4 corresponding to E4 is the complement of L3 . We can construct from A3 a DFA A4 accepting L4 .
Let E5 be the expression E2 _ E4 . Clearly, L5 = L2 L4 . We can construct from A2 and A4
a DFA A5 to accept L5 .
Finally, let E6 (this is E ) be the expression E1 ^ E5 . Then L6 = L1 \ L5 . Again, we can
construct from A1 and A5 a DFA A6 to accept L6 .
Now consider the expression 9zE6 . Call this expression E7 . Then L7 = f(x y w)j9z such that(x y w z) 2 L6 g
+
Note that now L7
3 , where alphabet 3 = f(s1 s2 s3 )jsi 2 f0 1gg. We can construct an
NFA B7 to accept L7 .
Note that the input to B7 has only three tracks (corresponding to x y w)  the track corresponding to z is not part of the input. B7 is constructed from A6 . B7 \simulates" the
computation of A6 , but \guesses" (using nondeterminism) the bits of z as it reads the input.
B7 is then converted to a DFA A7 .
We can do the same thing for the expression E8 = 9w9zE . We can construct from A7 a DFA
A8 to accept L8 +, where now 2 = f(s1 s2)jsi 2 f0 1gg. For E9 = 8y9w9zE , we note
2
+
that this expression is equivalent to E9 = :9y:(9w9zE ). The language L9
1 , where
1 = f(s1 )js1 2 f0 1gg. We can construct a DFA A9 to accept L9 as follows: Construct a DFA B8 from A8 that accepts the complement of L8 . Thus B8 accepts the
language corresponding to :(9w9zE ).
Next construct from B8 a DFA C8 to accept the language corresponding to 9y:(9w9zE )
(we already know how to do this!). c 6 O.H.Ibarra Then construct from C8 a DFA A9 to accept the complement of the language accepted
by C8 . Thus A9 accepts the language corresponding to the expression :9y:(9w9zE ).
Finally, we construct from A9 a DFA A10 to accept the language L10 corresponding to the
+
formula F = 9x8y9w9zE . Note that L10
0 , where 0 = f()g, where () is a symbol
corresponding to an empty tuple. It is easily veri ed that A10 accepts any input if and only
if F is true. So all we need to check is whether A10 accepts the null string .
The constructions in the example above can be formalized to prove the claim. 1.1 Machine Models
We can classify algorithms in terms of thier resource (memory) requirements:
A. Fixed memory procedures.
Some procedures use only a xed amount of memory, independent of the size of the input
(i.e. the device carrying out such a procedure only needs a xed amount of memory).
For example, the addition of two integers represented in binary.
(a1 ,b1 ), (a2 ,b2 ),....,(an,bn )  Device doing
Addition  s1,s2 ,...,sn This device needs only 1 bit of memory for the carry bit.
B. Unbounded memory procedures.
Some procedures use memory that grows with the input size (i.e. the device carrying out such
a procedure needs an unbounded memory).
For example, the multiplication of two integers represented in binary.
(a1 ,b1 ), (a2 ,b2 ),....,(an,bn )  Device doing Multiplication  p1,p2,...,pn Figure 1.1:
This device needs memory that is a function of n.
Access to memory may be restricted or unrestricted. Some examples of restricted memory
access machines are pushdown stacks, queues, and dequeues. Some examples of unrestricted
memory access machines are disks, drums, tape units, and RAMs. c 7 O.H.Ibarra C. A general machine.
We now introduce a general model of computation. By restricting this machine in various
ways, we get machines such as the Turing machine, pushdown automaton, and nite automaton. For a general model, we need to have the following components: {
{
{
{ Input/Output
Potentially unbounded memory
CPU
Program
Input tape (2way read only tape)
c a1 a2 a3 ... an $ 2way read only head
Finite State
Control
{q1 , q 2, ..., q k} 1way write only head Output tape( 1way write only tape)
Auxiliary
Memory Figure 1.2: A General Model of a Computing Machine
The above gure shows the components of such a model. The tapes are divided into cells
which the control can read one at a time. The nite state control represents the CPU and
program components. This control has access to the input tape which holds the input to
the machine. This tape is a read only two way tape. In other words, the head can move in
both directions  left and right or remain at the same cell, but it cannot modify the contents
of any cell. The nite control also has access to the an auxillary memory which models the
unbounded memory of the machine. This memory may be organized in various ways such
as a pushdown stack or a queue or in nite length tapes depending upon the machine. The
control also has access to an output tape onto which it writes the output. This output tape
is one way, i.e. the control cannot move back on this tape once it has written a cell  it only
moves forward or remains at the same cell.
The machine shown above can be described as M =< Q
; / $ q0 >. Where
c
Q is the set of states of the nite control, c 8 O.H.Ibarra is the input alphabet,
is the output alphabet,
; is the memory or worktape alphabet,
c
/ is the left end marker for the input,
$ is the right end marker for the input,
q0 is the initial state in which the machine begins the computation,
is the program of the machine consisting of a nite set of rules.
Programming this machine is done by using rules of the following form.
(q a ) ! (q0 d 0 b) q 2 fq1 q2 ::: qk g is the current state.
a 2 is the current symbol under the ROH.
is a \local portion" of the auxiliary memory.
q0 2 fq1 q2 ::: qk g is the next state.
d is the direction of the ROH movement (;1, 0, or +1).
0 is the change in the local portion of the auxiliary memory. b is the output symbol written on the WOH (could be ).
For example, if the auxiliary memory is a pushdown stack then machine is a pushdown
automaton, and a possible rule is (q5 0 a) ! (q3 +1 pop ). 1.2 Examples of Machines
A simpli ed version of the general machine is shown in the gure below. We have combined the
input tape and the memory to form an in nite tape in one direction which initially holds the input.
The input tape is now a 2way read/write tape.
Let us now consider some speci c examples of machines.
Example 1 If the machine has no memory and a 1way input, then it is called a 1way Finite
Automaton or 1FA.
Finite automata are simple machines that can recognize Regualar Expressions. The working
of Finite Automata can also be described by state diagrams. Consider the following example of
a binary adder which adds two binary numbers. The numbers are represented in binary and are
given to machine as input. The machine works correctly only if the digits are in least signi cant c 9 O.H.Ibarra c
/ a1 a2 a3 ... $ 2way read/write head
Finite State
Control
{q1 , q 2, ..., q k} 1way write only head Output tape( 1way write only tape) Figure 1.3: A Simpler Model of a Computing Machine c
/ a1 a2 a3 ... $ 1way readonly head
Finite State
Control
{q1 , q 2, ..., q k} 1way write only head Output tape( 1way write only tape) Figure 1.4: A Model of a 1way Finite Automaton. End markers are not needed for oneway FA's. c 10 O.H.Ibarra bit rst order. The input alphabet consists of pairs of bits, one for each number:
P
= f(0 0) (0 1) (1 0) (1 1)g
The output alphabet consists of bits:
= f0 1g. Finite automata can be described by state diagrams such as the one shown in Figure 1.5
for the binary adder.
(1,1)/0
(0,0)/0
(0,1)/1
(1,0)/1 q0 (1,0)/0
(0,1)/0
(1,1)/1 q1
(0,0)/1 Start Figure 1.5: State Diagram for a Binary Adder
Some FA have no output, and are called pattern recognizers. Given an input string the machine
will enter an accept or a reject state after processing the input string, depending on whether the
input string is part of the language that the machine recognizes. An example is a machine which
is given a sequence of Heads (H ) and Tails, (T ) and it accepts the string only if every Head is
followed by at least 2 Tails. Figure 1.6 shows the state diagram for this machine. T
T
q0 T q1 H q2 T q3 H H Start
dead
T,H
Figure 1.6: State Diagram of a Pattern Recognizer
P The machine can be described as A =< Q
q0 F >, where the input alphabet of the
P
machine, = fH T g,
Q = fq0 q1 q2 q3 deadg is the set of states, q0 is the initial state, F = fq1g is the set of nal or
accepting states and the transition function, is de ned by the diagram. c O.H.Ibarra 11 1.3 Nondeterminism
Nondeterminism is the mathematical model for parallel computation. In terms of machines, nondeterminism introduces choices. The translation rules, , or program of the machine describes the
next state of the machine given the current state. In a deterministic machine there is no choice of
the next state. For a nondeterministic machine, in contrast, the machine can have more than one
choice of the next state. Due to this choice, it is not clear which of the multiple paths available will
be followed in any given execution of the machine. The machine can potentially produce di erent
results in separate executions starting with the same input.
For nondeterministic machines that accept or reject the input strings, a string is said to be
accepted by the machine if there is at least one execution that accepts the string. Therefore, if
a given execution of a nondeterministic machine results in the rejection of the input, it is not
necessarily true that the string is rejected by the machine. 1.3.1 Finite Automata
The simplest machines are those with no memory (actually, they have a xed amount of memory
which cannot be increased, i.e., the memory is independent of the input size). They are called
nitestate machines. A special case is a nite automaton (FA). FA's are pattern recognizers.
Figure 1.7 shows the model. An FA can be formally described as M =< Q
q0 F >, where Q is the nite set of states,
is the input alphabet,
is the transition function which de nes the actions of the FA, q0 is the initial state and
F is the set of accepting or nal states.
The input head can be either 1way or 2way, and the transition function may (in the case of
nondeterministic) or may not (in the case of deterministic) allow multiple moves from a state on
the same input. These two options result in four varieties of FA depending on whether they are
1way or 2way and deterministic or nondeterministic. (The 2way varieties are provided input
delimiters.) The four varieties are called 1DFA, 1NFA, 2DFA, 2NFA. All four varieties of FA are
equivalent in their recognizing power. In other words, any language that can be accepted by an
FA of any variety, can also be accepted by some FA of any of the other varieties. Therefore we
see that for memoryless machines, having a 2way head or nondeterminism does not result in
any gains. The languages accepted by FA are known as Regular Sets and can be described using
Regular Expressions. It is important to note that even though all four varieties of FA are equivalent c 12 O.H.Ibarra a1 a2 a3 a4 ... 1way readonly head
Finite State
Control
{q1 , q 2, ..., q k} Figure 1.7: A Model of a Finite Automaton
in terms of the language that they can recognize, some are more e cient (\succinct") than others.
In particular, it can be shown that there are certain languages that can be accepted by an O(n)
state 1NFA but requires at least O(2n ) states by any 1DFA accepting the same language.
Consider the following set of languages. For every positive integer n, let Ln = fx1y j x in f0 1g jyj =
n ; 1g.
Ln can be accepted by a 1NFA with n + 1 states as follows 0,1
1 0,1 0,1 .... 0,1 n1 states Start
For example, L2 can be accepted by the following 1NFA. 0,1
1 0,1 Start
Ln can be accepted by a 2DFA with O(n) states. c 13 O.H.Ibarra Claim: Any 1DFA accepting Ln needs at least 2n states.
Proof...
View
Full Document
 Winter '08
 Ibarra,O
 Turing Machines

Click to edit the document details