Computability and Complexity

Computability and Complexity Turing Machines CAS 705 Ryszard Janicki Department of Computing and Software McMaster University Hamilton, Ontario, Canada janicki@mcmaster.ca Ryszard Janicki Computability and Complexity 1 / 56

Introduction It is named after Alan Turing, who invented them in 1936 Turing machines can compute any function normally considered computable Turing Machines were invented long before real computers appeared They came at a time when mathematicians were trying to come to grips with the notion of eective computation Several alternative formalisms evolved in the attempt to nail down eectively computable Ryszard Janicki Computability and Complexity 2 / 56

Introduction The three most popular alternative formalisms are: Turing machines (Alan Turing) machines approach Post systems (Emil Post) rewriting systems approach Recursive functions (Kurt Gödel, Alonzo Church) calculation with numbers approach Ryszard Janicki Computability and Complexity 3 / 56

Introduction The one that is the more popular, as some claim (falsely) it most closely resembles a modern computer is the Turing machine Besides the most know model, there are also many variations nondeterministic multitape multidimensional tape two-way innite tapes They can all simulate one another Ryszard Janicki Computability and Complexity 4 / 56

Introduction Church's thesis (or the Church-Turing thesis) It is not a theorem, but rather a declaration All these (including Turing Machines) formalisms capture precisely our intuition about what it means to be eectively computable in principle, no more and no less Very likely that the most compelling development leading to the acceptance of Church's thesis was the Turing machine (model considered readily programmable) Ryszard Janicki Computability and Complexity 5 / 56

Informal Description of Turing Machines We will be considering a deterministic, one-tape Turing machine This is the standard model There are many variations, apparently more powerful or less powerful but in reality not TM has a nite set of states Q a semi-innite tape that is delimited on the left end by an endmarker and is innite to the right a head that can move left and right over the tape, reading and writing symbols Ryszard Janicki Computability and Complexity 6 / 56

Informal Description of Turing Machines a b b a b a.... Q Two way, read/write Turing Machine (TM) can both write on the tape and read from it. The read-write head can move both to the left and to the right. The special states for rejecting and accepting take eect immediately. There are many slightly dierent denitions of Turing Machines. Ryszard Janicki Computability and Complexity 7 / 56

Formal Description of Turing Machines Denition (One of many) A deterministic one-tape Turing machine is a 7-tuple M = (Q, Σ, Γ, δ, s, t, r), where Q is a nite set (the states); Σ is a nite set (the input alphabet); Γ is a nite set (the tape alphabet) containing Σ as a subset; [ Γ \ Σ, the blank symbol ( [ ε); often we assume, the left endmarker, belongs to Γ \ Σ; δ : Q Γ Q Γ {L, R}, the transition function; Sometimes δ is dened as δ : Q Γ Q Γ {L, S, R}. Sometimes δ is dened as δ : Q Γ Q Γ {L, R}, where Q = Q \ {r, t}. s Q, the start state; t Q, the accept state; and r Q, the reject state, r t. Ryszard Janicki Computability and Complexity 8 / 56

Formal Description of Turing Machines Intuitively, δ (p, a) = (q, b, d) means, When in state p scanning symbol a, write b on that tape cell, move the head in direction d, and enter state q. The symbols L and R stand for left and right, respectively (and S means `stay in the same place') Often we assume that p Q. q Q. δ (p, ) = (q,, R) We refer to the state set and transition function collectively as the nite control Ryszard Janicki Computability and Complexity 9 / 56

Formal Description of Turing Machines Example (with ) We are considering a TM that accepts the non-context-free set {a n b n c n n 0} Q = {s, q 1,, q 10, t, r} Σ = {a, b, c} Γ = Σ {, [, } is just an extra useful symbol in the tape alphabet (NOT a special symbol) Ryszard Janicki Computability and Complexity 10 / 56

Formal Description of Turing Machines Ryszard Janicki Computability and Complexity 11 / 56

Congurations Formally: Γ Q Γ uaq i bv uq j acv δ(q i, b) = (q j, c, L) uaq i bv uacq j v δ(q i, b) = (q j, c, R) means `yields'. Special cases: Left-hand end: q i bv q j cv for L (if no ) Right-hand end: uaq i uaq i [ Ryszard Janicki Computability and Complexity 12 / 56

Special Congurations start conguration: sw or sw accepting conguration: utv rejecting conguration: ur v halting conguration: accepting or rejecting Turing Machine may not halt at all! It may run forever. Ryszard Janicki Computability and Complexity 13 / 56

Acceptance, Looping, Deciding A Turing machine M accepts input w if there is a sequence of congurations c 1, c 2,..., c k such that: 1 c 1 = sw 2 c i c i+1 for i = 1,..., k 1 3 c k is an accepting conguration. L(M) = {w M accepts w} L(M) is the language recognized by M. Turing machine loops if it neither accepts nor rejects some input. Turing machine decides if it does not loop. Ryszard Janicki Computability and Complexity 14 / 56

Turing Machines and Languages Denition A language L is Turing-recognizable if some Turing machine recognizes it. In this case, if x L, M accepts it (and stops), if x / L, M either rejects it or loops, so we do not really know if x L or not. Turing-recognizable recursively enumerable. Denition A language L is Turing-decidable if some Turing machine decides it. In this case, if x L, M accepts it (and stops), if x / L, M either rejects it (and stops). Turing-decidable recursive. Ryszard Janicki Computability and Complexity 15 / 56

Example The Turing machine described below decides L = {0 2n n 0} Idea: On input string w: 1 Sweep left to right across the tape, crossing o every other 0. 2 if in stage 1, the tape contained a single 0, accept. 3 If in stage 1 the tape contained more than a single 0 and the number of 0's was odd, reject. 4 Return the head to the left-hand end of the tape. 5 Go to stage 1. Ryszard Janicki Computability and Complexity 16 / 56

Example (M decides L = {0 2n n 0}, graphical) Ryszard Janicki Computability and Complexity 17 / 56

Functions vs Languages Turing Machines dened as in this Lecture Notes can at best accept languages, i.e. sets of sequences, so how one can say that they can compute anything? God made the natural numbers; all else is the work of man - Leopold Kronecker 1890. We can really eectively process only natural numbers! Unary notation: 0 = ε, 1 = 0, 2 = 00, 3 = 000,..., n = 0 n, etc. Note that for every function f : IN IN, we can write : f (k) = n 0 k #0 n. Similarly for every function g : IN IN IN, we can write : g(i, j) = n 0 i 10 j #0 n, etc. Each function f : IN IN can uniquely be represented by a language L f = {0 k #0 f (k) k IN}. Similarly each function g : IN IN IN can uniquely be represented by a language L g = {0 i 10 j #0 g(i,j) i, j IN}. Ryszard Janicki Computability and Complexity 18 / 56

Turing Machines that dene functions explicitly There is no real dierence between computing functions and recognizing languages. Languages are more general and often easier to handle. Nevertheless, we Turing machines can also dene functions explicitly, without using the concept of a language. Consider a standard deterministic single tape Turing Machine M = (Q, Σ, Γ, δ, s, t, r) with Σ = {0} or Σ = {0, 1}. We say that M computes the function f : IN IN if for each input 0 k on the tape, it reach the accept state t, when the tape contains the string 0 f (k). We say that M computes the function g : IN IN IN if for each input 0 i 10 j on the tape, it reach the accept state t, when the tape contains the string 0 g(i,j), etc. Ryszard Janicki Computability and Complexity 19 / 56

Multitape Turing Machines Several tapes instead of just one Each tape has its own head for reading and writing Initially the input appears on tape 1, and others start out blank The transition function is changed to allow for reading, writing and moving the heads on all tapes simultaneously, i.e. δ : Q Γ k A (Γ {L, R}) k where k is the number of tapes. Example: δ(q, a 1, a 2, a 3 ) = (p, (b 1, R), (b 2, L), (b 3, R)). Conguration and the `yields' relation have to be changed accordingly. Two Turing machines are equivalent if they recognize/decide the same language (compute the same function). Ryszard Janicki Computability and Complexity 20 / 56

Theorem Every multitape Turing machine has an equivalent single tape Turing Machine. Proof. (sketch). Ryszard Janicki Computability and Complexity 21 / 56

Idea of an algorithm is the following: say input is x = a 1... a k, and a single tape machine is S 1 S puts its tape into: # a 1 a 2... a k # # #... # }{{} [ [ k 1 2 To simulate a single move, S scans its tape from rst #, to (k + 1) th #, in order to determine the symbol under the virtual heads. Then S make a second pass to update the tapes according to M's transition function. 3 If S moves one of the virtual heads to the right onto a #, M has moved the corresponding head to next unused blank. So S writes blank and shifts the tape from this cell until the last #, one unit to the right. Theorem The simulation from the previous theorem is polynomial time. [ Ryszard Janicki Computability and Complexity 22 / 56

Nondeterministic Turing Machines - Denition The only dierence is that the transition function δ is of the following form: δ : Q Γ 2 Q Γ {L,R} Ryszard Janicki Computability and Complexity 23 / 56

Nondeterministic vs Deterministic Turing Machines Theorem Every non-deterministic Turing machine has an equivalent deterministic Turing machine. Proof. (idea). Depth-rst search does not work for innite trees. In this case the simulation time is exponential. Ryszard Janicki Computability and Complexity 24 / 56

Denition A non-deterministic Turing machine S is said to accept/recognize a language L, if for all w L, at least one of the possible congurations accepts w. There may be branches that lead to non accepting congurations, some may put the machine into an innite loop. But these are irrelevant for acceptance. Angelic non-determinism/semantics. Denition A non-deterministic Turing machine S is said to decide a language L, if for all w L, there is a path that leads to either acceptance or rejection. Angelic non-determinism/semantics again. However it can be replaced by `demonic' in this case. Ryszard Janicki Computability and Complexity 25 / 56

Enumerators Consider a multitape Turing machine M that uses one tape as an output tape, on which a symbol, once written, can never be changed, and whose tape symbol never moves left. On the input tape, M writes strings over Σ, separated by # / Σ. E(M), the language enumerated by M, is the set of w Σ such that w is eventually printed between a pair of #'s on the output tape. Note that unless M runs forever, E(M) is nite. No particular order is required on the output tape, and repetitions are allowed. Theorem The language L is Turing-recognizable (or recursively-enumerable) some enumerator enumerates it. Ryszard Janicki Computability and Complexity 26 / 56

Random Access Machines - Motivation How can one claim that Turing machines can simulate any program in any programming language, including those that have not been invented yet? Every existing hardware (except quantum computers) is a version of von Neumann machine, so it the input must an executable programs (or plain data). Hence it suces to have a mathematical model of executable programs (they all have similar structures) and next to show that Turing machines can simulate models of all executable programs. This mathematical model of executable programs is called Random Access Machine Programs or RAM-programs. It was proved about ten years ago that quantum computers also can be simulated by Turing machines, however some say that this proof used wrong assumptions. We will argue that RAM-programs can be simulated by Turing machines (and vice versa). Formal proofs of this can be found in some older textbooks or scientic literature (from fties and sixties). Ryszard Janicki Computability and Complexity 27 / 56

Von Neumann Machine (1) Ryszard Janicki Computability and Complexity 28 / 56

Von Neumann Machine (2) Ryszard Janicki Computability and Complexity 29 / 56

Random Access Machines Denition A random access machine (RAM) model is a one-accumulator computer in which instructions are not permitted to modify themselves. Each register can hold an integer of arbitrary size. Memory is innite. IMPORTANT: One random machine denes only one specic program. Ryszard Janicki Computability and Complexity 30 / 56

RAM - Denition x 1 x 2 x n Read-Only Input Tape Accumulator - r 0 r 1 Location Counter Program r 2 Memory (infinite) y 1 y 2 Ryszard Janicki Fig. 48 Computability and Complexity 31 / 56

Assumptions (can be changed) The size of the problem is small enough to t in the main memory of a computer. The integers used in the computation are small enough to t in one computer word. The program for a RAM is not stored in the memory, therefore program does not modify itself. All computation takes place in the rst register r 0 called accumulator. Ryszard Janicki Computability and Complexity 32 / 56

Sample of Instructions RAM instructions correspond to the instructions used in executable Sample of instructions: programs and resemble Assembler instructions. Operation Code LOAD STORE ADD SUB MULT DIV READ WRITE JUMP JGTZ JZERO Address OPERAND OPERAND OPERAND OPERAND OPERAND OPERAND OPERAND OPERAND LABEL LABEL LABEL HALT Notation for Operands: Ryszard Janicki Computability and Complexity 33 / 56

Operands and Memory Maps An operand can be one of the following 1 = i indicates i itself. 2 A non-negative integer i, indicating the content of register i. 3 i, indicating indirect addressing. That is the contents of register j, where j is the integer found in register i. If j < 0, the machine halts. A memory map is a function c : {0, 1, 2,...} Integers, where c(i) is the integer stored in register i. Ryszard Janicki Computability and Complexity 34 / 56

Program Execution and Values Operands The program is executed as follows: A memory map is a function c : {0, 1, 2,...} Integers, where c(i) is the integer stored in register i. Initially, c(i) = 0, i 0, location counter is set to the rst instruction in P, the output tape is blank. After execution of the k th instruction in P, the location counter is automatically set to k + 1, unless the k th instruction is JUMP, HALT, JGTZ, or JZERO. To specify the meaning of an instruction we dene v(a), the value of operand a, as follows: v(= i) = i, v(i) = c(i), v(i) = c(c(i)) Instructions not dened, such as for example STORE= i are usually considered as equivalent to HALT. Likewise, division by zero halts the machine. Ryszard Janicki Computability and Complexity 35 / 56

Meaning of RAM Instructions The operand a is either = i, i, or i. x (or oor) denotes the greatest integer equal to or less than x. Ryszard Janicki Computability and Complexity 36 / 56

What RAM Program Does? 1 If x 1,..., x n are inputs, y is the rst output when RAM halts, then RAM computes: f RAM (x 1,..., x n ) = y 2 If a sequence a 1... a n # in on rst n + 1 cells of the input tape (# is just an end-marker), then a RAM program accepts a 1... a n it it reads a 1... a n #, writes 'YES' in the rst cell of the output tape, end halts. Ryszard Janicki Computability and Complexity 37 / 56

Example { n n n 0, n Integers Consider the function: f (n) = 0 otherwise possible pseudocode is: A The loop for is more convenient but while is easier to translate into RAM. Ryszard Janicki Computability and Complexity 38 / 56

RAM program Corresponding pseudocode program Ryszard Janicki Computability and Complexity 39 / 56

Example Consider the language {w w {1, 2} # 1 (w) = # b (w)}, and assume that 0 is endmarker. A possible pseudocode is: Ryszard Janicki Computability and Complexity 40 / 56

RAM program Corresponding pseudocode program Ryszard Janicki Computability and Complexity 41 / 56

Turing Machines and RAM Programs Theorem Every Turing machine can be simulated by a RAM program. Proof. (sketch). Registers of RAM simulate tape. Ryszard Janicki Computability and Complexity 42 / 56

RAM Programs and Turing Machines Theorem Let L be a language accepted by a RAM program. There is a Turing machine that accepts L. Proof. (sketch). We represent all the RAM registers not holding 0 as below: The tape consists of a sequence of pairs (i j, c j ) written in binary with no leading 0's and separated by marker symbols. For each j, c j is the content of RAM register j, i.e. c j = c(j). The contents of the RAM accumulator is stored in binary on a second tape, and the third tape is used for a scratch memory. Two other tapes serve to hold the input and the output of the RAM. Each step of the RAM program is represented by a nite set of states of the Turing machine. Ryszard Janicki Computability and Complexity 43 / 56

We shall not describe the simulation of an arbitrary RAM instruction, but shall consider only the instructions ADD 20 and STORE 30, which should make the ideas clear. For ADD 20, we can design the Turing machine to do the following: 1 Seach tape 1 for any entry for RAM register 20, i.e., a sequence ##10100#. If found, place the integer following, which will be the contents of register 20, on tape 3. If not found, then halt. The contents of register 20 is 0 and thus the indirect addressing cannot be done. 2 Look on tape 1 for an entry for The RAM register whose number is stored on tape 3. If found, copy the content of that register onto tape 3. If not found, place 0 there. 3 Add the number placed on tape 3 in step 2 to the content of the accumulator, which is held on tape 2. We assume Turing machines can do additions. Ryszard Janicki Computability and Complexity 44 / 56

To simulate the instruction STORE 30, we can design the Turing machine to do the following: 1 Search for an entry fro RAM register 30, i.e., ##11110#. 2 If found, copy everything to the right of ##11110#, except for the integer immediately following (the old contents of register 30), onto tape 3. Then copy the contents of the accumulator (tape 2) immediately to the right of ##11110# and follow it by the string copied onto tape 3. If no entry for register 30 was found on tape 1, instead go to the leftmost blank, print 11110#, followed by the contents of the accumulator, followed by ##. Ryszard Janicki Computability and Complexity 45 / 56

Computational Complexity of RAM Programs Uniform Cost Criterion: Each instruction requires one unit of time and each register requires one unit of space. Logarithmic Cost Criterion: { FLOOR(log2 i ) + 1 i 0 l(i) = 1 i = 0 Logarithmic Cost of RAM Operands: t(a) - time cost of a Ryszard Janicki Computability and Complexity 46 / 56

Computational Complexity of RAM Programs Logarithmic Cost of RAM Instructions: t(a) - time cost of operand a, b denotes a label Ryszard Janicki Computability and Complexity 47 / 56

Complexity of our program for n n. Time: it is dominated by the loop with MULT instruction, so under uniform cost time is O(n) (MULT is executed n 1 times). Under the logarithmic cost, the cost of executing the i th MULT is: l(n i ) + l(n) (i + 1) log 2 n, so the total cost is n 1 (i + 1) log n = O(n 2 log n). i=1 Space: Uniform cost is O(1). Logarithmic cost: the largest integer stored is l(n n ) n log n, so it is O(n log n). Ryszard Janicki Computability and Complexity 48 / 56

Random Access Stored Program Machine (RASP) 1 The instruction set for a RASP is identical to that for a RAM, except indirect accessing is not permitted since it is not needed. 2 A RASP can simulate indirect addressing by the modication of instructions during program execution. 3 The program of a RASP is assumed to be in the registers of the memory. 4 Each RASP instruction occupies 2 consecutive memory registers. The 1 st register holds an encoding; the 2 nd register holds the address. Ryszard Janicki Computability and Complexity 49 / 56

Possible code for RASP instructions For example, the instruction LOAD =32 would be stored with 2 in one register and 32 in the following register. Ryszard Janicki Computability and Complexity 50 / 56

The state of a RASP can be represented by: 1 The memory map c, where c(i), i 0, is the content of register. 2 The location counter, which indicates the rst of the two consecutive memory registers from which the current instruction is to be taken. Initially, the location counter is set at some specied register. The initial content of the memory registers are not all 0, since the program has been loaded into the memory at the start. All but a nite number of the registers contain 0 at the start and the accumulator contains 0. Location counter is set at some species register. After each instruction is executed, the location counter is increased by 2, except in the case of JUMP i, JGTZ i, or JZERO i, in which case the location counter is set to i. The time and/or space complexity can by dened in much the same manner as that of a RAM program. In the case of logarithmic cost we must also change for accessing the instruction itself, which costs l(lc). Ryszard Janicki Computability and Complexity 51 / 56

RAM vs RASP Theorem If costs of instructions are either uniform or logarithmic, for every RAM program of time complexity T (n) there is a constant C such that there is an equivalent RASP program of time complexity CT (n). Theorem If costs of instructions are either uniform or logarithmic, for every RASP program of time complexity T (n) there is a constant C such that there is an equivalent RAM program of time complexity at most CT (n). The similar theorems hold for space complexity. RAM and RASP can be considered as completely equivalent. Ryszard Janicki Computability and Complexity 52 / 56

RAM, RASP and Turing Machines again Theorem Every TM can be simulated by RAM in O(T (n)) time under uniform cost or in O(T (n) log T (n)) under logarithmic cost. Proof. Registers of RAM simulate tape. Theorem RAM's cannot be simulated by TM in polynomial time under uniform cost, they can be simulated in exponential time. Proof. Under the uniform cost an n-step RAM program can compute number as high as 2 2n, which requires 2 n TM cells just to store and read. Ryszard Janicki Computability and Complexity 53 / 56

RAM, RASP and Turing Machines again Theorem The RAM and RASP under logarithmic cost and Turing machines are all polynomially related. Theorem RAM and high level programming languages are related by constants. Ryszard Janicki Computability and Complexity 54 / 56

Programs, RAM Programs and Turing Machines - Final Comment It can be shown that programs is high level programming languages that use standard programming constructions are linearly related. It can be shown that RAM programs and Turing machines are polynomially related. Hence the programs is high level programming languages that use standard programming constructions and Turing machines are polynomially related. Ryszard Janicki Computability and Complexity 55 / 56

This is almost the end of TURING MACHINES THEORY Ryszard Janicki Computability and Complexity 56 / 56