lec01-lexicalanalyzer

lec01-lexicalanalyzer - Lexical Analyzer Lexical Analyzer...

Info iconThis preview shows pages 1–8. Sign up to view the full content.

View Full Document Right Arrow Icon
CS440 Compilers 1 Lexical Analyzer Lexical Analyzer reads the source program character by character to produce tokens. Normally a lexical analyzer doesn’t return a list of tokens at one shot, it returns a token when the parser asks a token from it. Lexical Analyzer Parser source program token get next token
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
CS440 Compilers 2 Token Token represents a set of strings described by a pattern. Identifier represents a set of strings which start with a letter continues with letters and digits The actual string (newval) is called as lexeme . Tokens: identifier, number, addop, delimeter, … Since a token can represent more than one lexeme, additional information should be held for that specific lexeme. This additional information is called as the attribute of the token. For simplicity, a token may have a single attribute which holds the required information for that token. For identifiers, this attribute a pointer to the symbol table, and the symbol table holds the actual attributes for that token. Some attributes: <id,attr> where attr is pointer to the symbol table <assgop,_> no attribute is needed (if there is only one assignment operator) <num,val> where val is the actual value of the number. Token type and its attribute uniquely identifies a lexeme. Regular expressions are widely used to specify patterns.
Background image of page 2
CS440 Compilers 3 Regular Expressions We use regular expressions to describe tokens of a programming language. A regular expression is built up of simpler regular expressions (using defining rules) Each regular expression denotes a language. A language denoted by a regular expression is called as a regular set .
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
CS440 Compilers 4 Regular Expressions (Rules) Regular expressions over alphabet Σ Reg. Expr Language it denotes ε { ε } a Σ {a} (r 1 ) | (r 2 ) L(r 1 ) L(r 2 ) (r 1 ) (r 2 ) L(r 1 ) L(r 2 ) (r) * (L(r)) * (r) L(r) (r) + = (r)(r) * (r)? = (r) | ε
Background image of page 4
CS440 Compilers 5 Regular Expressions (cont.) We may remove parentheses by using precedence rules. * highest concatenation next | lowest ab * |c means (a(b) * )|(c) Ex: Σ = {0,1} 0|1 => {0,1} (0|1)(0|1) => {00,01,10,11} 0 * => { ε ,0,00,000,0000,. ...} (0|1) * => all strings with 0 and 1, including the empty string
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
CS440 Compilers 6 Regular Definitions To write regular expression for some languages can be difficult, because their regular expressions can be quite complex. In those cases, we may use regular definitions . We can give names to regular expressions, and we can use these names as symbols to define other regular expressions. A regular definition is a sequence of the definitions of the form: d 1 r 1 where d i is a distinct name and d 2 r 2 r i is a regular expression over symbols in . Σ∪ {d 1 ,d 2 ,...,d i-1 } d n r n basic symbols previously defined names
Background image of page 6
CS440 Compilers 7 Regular Definitions (cont.) Ex: Identifiers in Pascal letter
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 8
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 05/04/2010 for the course CS 101010 taught by Professor Zaki during the Spring '10 term at DeVry Cleveland D..

Page1 / 35

lec01-lexicalanalyzer - Lexical Analyzer Lexical Analyzer...

This preview shows document pages 1 - 8. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online