**Unformatted text preview: **Models of Computation
Jeff Erickson January 4, 2015 © Copyright 2014 Jeff Erickson. Last update January 4, 2015. This work may be freely copied and distributed in any medium.
It may not be sold for more than the actual cost of reproduction, storage, or transmittal.
This work is available under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
For license details, see .
For the most recent edition, see . I’m writing a book.
I’ve got the page numbers done,
so now I just have to fill in the rest.
— Stephen Wright About These Notes
These are lecture notes that I wrote for the course “Algorithms and Models of Computation”
at the University of Illinois, Urbana-Champaign for the first time in Fall 2014. This course is a
broad introduction to theoretical computer science, aimed at third-year computer science and
computer engineering majors, that covers both fundamental topics in algorithms, for which I
already have copious notes, and fundamental topics on formal languages and automata, for
which I wrote the notes you are reading now.
The most recent revision of these notes (or nearly so) is available online at .
illinois.edu/~jeffe/teaching/algorithms/, along with my algorithms notes and a near-complete
archive of past homeworks and exams from all my theoretical computer science classes. I plan to
revise and reorganize these whenever I teach this material, so you may find more recent versions
on the web page of whatever course I am currently teaching. About the Exercises
Each note ends with several exercises, many of which I used in homeworks, discussion sections,
or exams. ? Stars indicate more challenging problems (which I have not used in homeworks,
discussion sections, or exams). Many of these exercises were contributed by my amazing teaching
assistants:
Alex Steiger, Chao Xu, Connor Clark, Gail Steitz, Grant Czajkowski, Hsien-Chih
Chang, Junqing Deng, Nick Bachmair, and Tana Wattanawaroon
Please do not ask me for solutions to the exercises. If you are a student, seeing the solution
will rob you of the experience of solving the problem yourself, which is the only way to learn the
material. If you are an instructor, you shouldn’t ask your students to solve problems that you
can’t solve yourself. (I don’t always follow my own advice, so I’m sure some of the problems are
buggy.) Caveat Lector!
These notes are best viewed as an unfinished first draft. You should assume the notes
contain several major errors, in addition to the usual unending supply of typos, fencepost errors,
off-by-one errors, and brain farts. Before Fall 2014, I had not taught this material in more than
two decades. Moreover, the course itself is still very new—Lenny Pitt and I developed the
course and offered the first pilot in Spring 2014 (with Lenny presenting the formal language
material)—so even the choice of which material to emphasize, sketch, or exclude is still very
much in flux.
I would sincerely appreciate feedback of any kind, especially bug reports.
Thanks, and enjoy!
— Jeff ii Contents
1 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 Regular Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3 Finite-State Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4 Nondeterministic Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5 Context-Free Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6 Turing Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
7 Universal Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
8 Undecidability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
9 Nondeterministic Turing Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 iii Models of Computation Lecture 1: Strings [Fa’14] THOMAS GODFREY, a self-taught mathematician, great in his way, and afterward inventor of what is
now called Hadley’s Quadrant. But he knew little out of his way, and was not a pleasing companion;
as, like most great mathematicians I have met with, he expected universal precision in everything
said, or was forever denying or distinguishing upon trifles, to the disturbance of all conversation.
He soon left us.
— Benjamin Franklin, Memoirs, Part 1 (1771)
describing one of the founding members of the Junto
I hope the reader sees that the alphabet can be understood by any intelligent being who has any
one of the five senses left him,—by all rational men, that is, excepting the few eyeless deaf persons
who have lost both taste and smell in some complete paralysis. . . . Whales in the sea can telegraph
as well as senators on land, if they will only note the difference between long spoutings and short
ones. . . . A tired listener at church, by properly varying his long yawns and his short ones, may
express his opinion of the sermon to the opposite gallery before the sermon is done.
— Edward Everett Hale, “The Dot and Line Alphabet”, Altlantic Monthy (October 1858)
If indeed, as Hilbert asserted, mathematics is a meaningless game played with meaningless marks
on paper, the only mathematical experience to which we can refer is the making of marks on paper.
— Eric Temple Bell, The Queen of the Sciences (1931) 1 Strings Throughout this course, we will discuss dozens of algorithms and computational models that
manipulate sequences: one-dimensional arrays, linked lists, blocks of text, walks in graphs,
sequences of executed instructions, and so on. Ultimately the input and output of any algorithm
must be representable as a finite string of symbols—the raw contents of some contiguous portion
of the computer’s memory. Reasoning about computation requires reasoning about strings.
This note lists several formal definitions and formal induction proofs related to strings. These
definitions and proofs are intentionally much more detailed than normally used in practice—most
people’s intuition about strings is fairly accurate—but the extra precision is necessary for any
sort of formal proof. It may be helpful to think of this material as part of the “assembly language”
of theoretical computer science. We normally think about computation at a much higher level
of abstraction, but ultimately every argument must “compile” down to these (and similar)
definitions. 1.1 Definitions Fix an arbitrary finite set Σ called the alphabet; the elements of Σ are called symbols or
characters. As a notational convention, I will always use lower-case letters near the start of
the English alphabet (a, b, c, . . .) as symbol variables, and never as explicit symbols. For explicit
symbols, I will always use fixed-width upper-case letters (A, B, C, . . . ), digits (0, 1, 2, . . . ), or
other symbols (, $, #, •, . . . ) that are clearly distinguishable from variables.
A string (or word) over Σ is a finite sequence of zero or more symbols from Σ. Formally, a
string w over Σ is defined recursively as either
• the empty string, denoted by the Greek letter " (epsilon),or
• an ordered pair (a, x ), where a is a symbol in Σ and x is a string over Σ.
© Copyright 2014 Jeff Erickson.
This work is licensed under a Creative Commons License ( ).
Free distribution is strongly encouraged; commercial distribution is expressly forbidden.
See for the most recent revision. 1 Models of Computation Lecture 1: Strings [Fa’14] We normally write either a · x or simply ax to denote the ordered pair (a, x). Similarly, we
normally write explicit strings as sequences of symbols instead of nested ordered pairs; for
example, STRING is convenient shorthand for the formal expression (S, (T, (R, (I, (N, (G, ")))))).
As a notational convention, I will always use lower-case letters near the end of the alphabet
(. . . , w, x, y, z) to represent unknown strings, and SHOUTYMONOSPACEDTEXT to represent explicit
symbols and (non-empty) strings.
The set of all strings over Σ is denoted Σ∗ (pronounced “sigma star”). It is very important to
remember that every element of Σ∗ is a finite string, although Σ∗ itself is an infinite set containing
strings of every possible finite length.
The length |w | of a string w is the number of symbols in w, defined formally as follows:
¨
0
if w = ",
|w| :=
1 + |x| if w = ax.
For example, the string SEVEN has length 5. Although they are formally different objects, we do
not normally distinguish between symbols and strings of length 1.
The concatenation of two strings x and y, denoted either x • y or simply x y, is the
unique string containing the characters of x in order, followed by the characters in y in order.
For example, the string NOWHERE is the concatenation of the strings NOW and HERE; that is,
NOW • HERE = NOWHERE. (On the other hand, HERE • NOW = HERENOW.) Formally, concatenation is
defined recusively as follows:
¨
z
if w = "
w • z :=
a · (x • z) if w = ax
(Here I’m using a larger dot • to formally distinguish the operator that concatenates two arbitrary
strings from from the operator · that builds a string from a single character and a string.)
When we describe the concatenation of more than two strings, we normally omit all dots
and parentheses, writing wx yz instead of (w • (x • y)) • z, for example. This simplification is
justified by the fact (which we will prove shortly) that • is associative. 1.2 Induction on Strings Induction is the standard technique for proving statements about recursively defined objects.
Hopefully you are already comfortable proving statements about natural numbers via induction,
but induction actually a far more general technique. Several different variants of induction
can be used to prove statements about more general structures; here I describe the variant
that I recommend (and actually use in practice). This variant follows two primary design
considerations:
• The case structure of the proof should mirror the case structure of the recursive definition. For example, if you are proving something about all strings, your proof should have
two cases: Either w = ", or w = ax for some symbol a and string x.
• The inductive hypothesis should be as strong as possible. The (strong) inductive hypothesis for statements about natural numbers is always “Assume there is no counterexample k
such that k < n.” I recommend adopting a similar inductive hypothesis for strings: “Assume
there is no counterexample x such that |x| < |w|.” Then for the case w = a x, we have
|x| = |w| − 1 < |w| by definition of |w|, so the inductive hypothesis applies to x.
2 Models of Computation Lecture 1: Strings [Fa’14] Thus, string-induction proofs have the following boilerplate structure. Suppose we want to prove
that every string is perfectly cromulent, whatever that means. The white boxes hide additional
proof details that, among other things, depend on the precise definition of “perfectly cromulent”.
Proof: Let w be an arbitrary string.
Assume, for every string x such that |x| < |w|, that x is perfectly cromulent.
There are two cases to consider. • Suppose w = " . Therefore, w is perfectly cromulent. • Suppose w = a x for some symbol a and string x . The induction hypothesis implies that x is perfectly cromulent. Therefore, w is perfectly cromulent. In both cases, we conclude that w is perfectly cromulent. Here are three canonical examples of this proof structure. When developing proofs in this
style, I strongly recommend first mindlessly writing the green text (the boilerplate) with lots of
space for each case, then filling in the red text (the actual theorem and the induction hypothesis),
and only then starting to actually think.
Lemma 1.1. For every string w, we have w • " = w.
Proof: Let w be an arbitrary string. Assume that x • " = x for every string x such that |x| < |w|.
There are two cases to consider:
• Suppose w = ".
w•" ="•" because w = ", =" by definition of concatenation, =w because w = ". • Suppose w = ax for some symbol a and string x.
w • " = (a · x) • " because w = ax, = a · (x • ") by definition of concatenation, =a·x by the inductive hypothesis, =w because w = ax.
In both cases, we conclude that w • " = w.
Lemma 1.2. Concatenation adds length: |w • x| = |w| + |x| for all strings w and x. Proof: Let w and x be arbitrary strings. Assume that | y • x| = | y| + |x| for every string y such
that | y| < |w|. (Notice that we are using induction only on w, not on x.) There are two cases to
consider:
3 Models of Computation Lecture 1: Strings [Fa’14] • Suppose w = ".
|w • x| = |" • x| because w = "
by definition of | | = |x| |e| = 0 by definition of | | = |"| + |x| because w = " = |w| + |x| • Suppose w = a y for some symbol a and string y.
|w • x| = |a y • x| because w = a y = |a · ( y • x)| by definition of • = 1 + | y • x| by definition of | | = 1 + | y| + |x| by the inductive hypothesis = |a y| + |x| by definition of | |
because w = a y = |w| + |x| In both cases, we conclude that |w • x| = |w| + |x|. Lemma 1.3. Concatenation is associative: (w • x) • y = w • (x • y) for all strings w, x, and y.
Proof: Let w, x, and y be arbitrary strings. Assume that (z • x) • y = w • (x • y) for every string
z such that |z| < |w|. (Again, we are using induction only on w.) There are two cases to consider.
• Suppose w = ". (w • x) • y = (" • x) • y because w = " =x•y by definition of • = " • (x • y) by definition of • = w • (x • y) because w = " • Suppose w = az for some symbol a and some string z.
(w • x) • y = (az • x) • y because w = az = (a · (z • x)) • y by definition of • = a · (z • (x • y)) by the inductive hypothesis = a · ((z • x) • y) by definition of • = az • (x • y) by definition of • = w • (x • y) because w = az In both cases, we conclude that (w • x) • y = w • (x • y). This is not the only boilerplate that one can use for induction proofs on strings. For example,
we can modify the inductive case analysis using the following observation: A non-empty string w
is either a single symbol or the concatenation of two non-empty strings, which (by Lemma 1.2)
must be shorter than w. Here is a proof of Lemma 1.3 that uses this alternative recursive structure:
4 Models of Computation Lecture 1: Strings [Fa’14] Proof: Let w, x, and y be arbitrary strings. Assume that (z • x 0 ) • y 0 = z • (x 0 • y 0 ) for all strings
x 0 , y 0 , and z such that |z| < |w|. (We need a stronger induction hypothesis here than in the
previous proofs!) There are three cases to consider.
• Suppose w = ".
(w • x) • y = (" • x) • y because w = " =x•y by definition of • = " • (x • y) by definition of • = w • (x • y) because w = " • Suppose w is equal to some symbol a.
(w • x) • y = (a • x) • y because w = a = (a · x) • y because a • z = a · z by definition of • = a • (x • y) because a • z = a · z by definition of • = a · (x • y) by definition of • = w • (x • y) because w = a • Suppose w = uv for some nonempty strings u and v.
(w • x) • y = ((u • v) • x) • y because w = uv = (u • (v • x)) • y by the inductive hypothesis, because |u| < |w| = u • ((v • x) • y)
= u • (v • (x • y))
= (u • v) • (x • y)
= w • (x • y) by the inductive hypothesis, because |u| < |w| by the inductive hypothesis, because |v| < |w| by the inductive hypothesis, because |u| < |w|
because w = uv In both cases, we conclude that (w • x) • y = w • (x • y). 1.3 Indices, Substrings, and Subsequences For any string w and any integer 1 ≤ i ≤ |w|, the expression wi denotes the ith symbol in w,
counting from left to right. More formally, w i is recursively defined as follows:
(
a
if w = a x and i = 1
w i :=
x i−1 if w = a x and i > 1
As one might reasonably expect, w i is formally undefined if i < 1 or w = ", and therefore (by
induction) if i > |w|. The integer i is called the index of w i .
We sometimes write strings as a concatenation of their constituent symbols using this
subscript notation: w = w1 w2 · · · w|w| . While standard, this notation is slightly misleading, since
it incorrectly suggests that the string w contains at least three symbols, when in fact w could be a
single symbol or even the empty string.
In actual code, subscripts are usually expressed using the bracket notation w [i]. Brackets
were introduced as a typographical convention over a hundred years ago because subscripts and
5 Models of Computation Lecture 1: Strings [Fa’14] superscripts¹ were difficult or impossible to type.² We sometimes write strings as explicit arrays
w[1 .. n], with the understanding that n = |w|. Again, this notation is potentially misleading;
always remember that n might be zero; the string/array could be empty.
A substring of a string w is another string obtained from w by deleting zero or more symbols
from the beginning and from the end. Formally, a string y is a substring of w if and only if
there are strings x and z such that w = x yz. Extending the array notation for strings, we write
w [i .. j ] to denote the substring of w starting at w i and ending at w j . More formally, we define
¨
w[i .. j] := " if j < i, w i · w[i + 1 .. j] otherwise. A proper substring of w is any substring other than w itself. For example, LAUGH is a proper
substring of SLAUGHTER. Whenever y is a (proper) substring of w, we also call w a (proper)
superstring of y.
A prefix of w[1 .. n] is any substring of the form w[1 .. j]. Equivalently, a string p is a prefix
of another string w if and only if there is a string x such that px = w. A proper prefix of w is
any prefix except w itself. For example, DIE is a proper prefix of DIET.
Similarly, a suffix of w[1 .. n] is any substring of the form w[i .. n]. Equivalently, a string s is a
suffix of a string w if and only if there is a string x such that xs = w. A proper suffix of w is any
suffix except w itself. For example, YES is a proper suffix of EYES, and HE is both a proper prefix
and a proper suffix of HEADACHE.
A subsequence of a string w is a strong obtained by deleting zero or more symbols from
anywhere in w. More formally, z is a subsequence of w if and only if
• z = ", or
• w = a x for some symbol a and some string x such that z is a subsequence of x.
• w = a x and z = a y for some symbol a and some strings x and y, and y is a subsequence
of x.
A proper subsequence of w is any subsequence of w other than w itself. Whenever z is a (proper)
subsequence of w, we also call w a (proper) supersequence of z.
¹The same bracket notation is also used for bibliographic references, instead of the traditional footnote/endnote
superscripts, for exactly the same reasons.
²A typewriter is an obsolete mechanical device loosely resembling a computer keyboard. Pressing a key on a
typewriter moves a lever (called a “typebar”) that strikes a cloth ribbon full of ink against a piece of paper, leaving the
image of a single character. Many historians believe that the ordering of letters on modern keyboards (QWERTYUIOP)
evolved in the late 1800s, reaching its modern form on the 1874 Sholes & Glidden Type-WriterTM , in part to separate
many common letter pairs, to prevent typebars from jamming against each other; this is also why the keys on most
modern keyboards are arranged in a slanted grid. (The common folk theory that the ordering was deliberately
intended to slow down typists doesn’t withstand careful scrutiny.) A more recent theory suggests that the ordering
was influenced by telegraph³ operators, who found older alphabetic arrangements confusing, in part because of
ambiguities in American Morse Code.
³A telegraph is an obsolete electromechanical communication device consisting of an electrical circuit with a
switch at one end and an electromagnet at the other. The sending operator would press and release a key, closing and
opening the circuit, originally causing the electromagnet to push a stylus onto a ...

View
Full Document