Class CS544: 2 Scanning 1 Overview
Scanning is also called lexical analysis. This is because the scanner analyzes the input stream, consisting of lexical elements, called lexemes. The term lexeme comes from the field of linguistics, where it means the smallest unit of a language which has meaning. Scanning is, perhaps, the part of compilation that is most based upon formal mathematics and foundations of computer science. For this class, we are ignoring preprocessing as a phase in the compilation process. There are many language systems that apply one or more preprocessors to the program. Preprocessing techniques are basically a subset of the material we will cover; although preprocessors can present their own set of problems.
First program transformation
The scanner converts the source program's stream of lexemes into an equivalent representation of tokens. Tokens represent the lexeme and the context of the lexeme. For example, a typical token contains: The type of lexeme (e.g., IDENTIFIER) The lexeme's image ("aVariable") The lexeme's position in the input stream
The next part of the compiler, the parser, works on a stream of tokens.
Clean up the input
Scanners normalize the input stream. There is often unwanted text, such as comments, white space, and other lexemes that have nothing to contribute to the program translation. The scanner removes it, or puts it in some special form, so that the parser and does not have to attend to it.
Lexeme interpretation
One of the interesting decisions a compiler writer must make is what type of analysis and synthesis operations should go in which compiler component. For example, consider the following part of a program's source: -1.23. How many lexemes (and corresponding tokens) are there? Various answers, all correct, are 1, 2, and 4. Let's look a little further into this. If your scanner takes into account the structure of real and signed numbers, there is a single token that would represent a real number with a value of -1.23. Perhaps your scanner separates the sign from the number. Then you have two tokens. The first represents the minus sign. The second represents the positive real number 1.23. Maybe your scanner takes even smaller bites of the input. Then, you would have the four lexemes: - 1 . 23. The last approach makes for a simpler scanner. It does, however, put more of a burden upon the parser. You need to decide which is more appropriate. My preference is to use the last approach. This has a couple of benefits: The scanner is not responsible for determining the appropriate format for numbers. The scanner does not have to make a determination as to whether the text: 1-2.3 is an arithmetic expression or two numbers.
Theoretical foundations
There are two theoretical building blocks that have helped make scanner construction efficient and automatic: finite automata and regular expressions.
Regular expressions
Regular expressions give us a way to formally express the lexical structure of a language. From your foundations course you should recall that we can build regular expressions using the following rules.
Let be the input alphabet. If a then a is a regular expression (denoting the set containing a). If a and b are regular expressions, then: (a) is a regular expression. a | b is a regular expression. ab is a regular expression. a* is a regular expression.
Finite automata
Finite automata are a way of representing the structure of regular languages, and hence, regular expressions. There are type two of finite automata: deterministic (DFA) and non- deterministic (NFA). Both of these are used in the construction of many types of scanners. Even when scanners are written by hand, the state-based approach and results from finite automata are used.
Three important results
When building a scanner, you will deal with tables that represent states of finite automata, and operations of combining regular expressions. In order to make your scanner efficient, you will base your code (or the scanner generator will base it's generated code) on three particular results from theory: Kleene's construction that takes a DFA to regular expressions. This is important for ensuring that the DFA that you use actually represents the language you think it does. Thompson's construction that creates an NFA from a regular expression. This is the first step to converting the language of the regular expression grammar to an executable scanner. Subset construction that transforms an NFA to a corresponding DFA. The DFA is the final form we want to have for implementing the scanner since our programming languages are (usually) deterministic.
There is a fourth result that you use to minimalize the DFA that results from the application of subset construction. The result is Hopcroft's algorithm.
Example
Let's take an example through the complete cycle. We will start with a regular expression, R: c(c|d|u)*. If you think of c representing the set of letter characters, d representing the digits, and u representing the underscore character, then you can see that R describes the syntax for identifiers in many programming languages.
Thompson's construction
We need to construct an NFA from the regular expression. This is done in the following sequence: Create the trivial NFAs for each character, c, d, and u. Using the NFAs for regular expression operators (fig. 2.4 in the textbook), combine the simpler NFAs into more complex ones in the following order: c|d, c|d|u, (c|d|u)*, c(c|d|u)*.
Subset construction
Use the subset construction algorithm shown in fig. 2.6 of the text to create a DFA from the ...