{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

lecture03 - Lexi ca l Ana lysi s L ectur e 3 CS 143 Lecture...

Info iconThis preview shows pages 1–10. Sign up to view the full content.

View Full Document Right Arrow Icon
CS 143 Lecture 3 1 Lexical Analysis Lecture 3
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
CS 143 Lecture 3 2 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples of regular expressions
Background image of page 2
CS 143 Lecture 3 3 Lexical Analysis What do we want to do? Example: if (i == j) Z = 0; else Z = 1; The input is just a string of characters: \tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1; Goal: Partition input string into substrings Where the substrings are tokens
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
CS 143 Lecture 3 4 What’s a Token? A syntactic category In English: noun, verb, adjective, … In a programming language: Identifier, Integer, Keyword, Whitespace, …
Background image of page 4
CS 143 Lecture 3 5 Tokens Tokens correspond to sets of strings. Identifier: strings of letters or digits, starting with a letter Integer: a non-empty string of digits Keyword: “else” or “if” or “begin” or … Whitespace: a non-empty sequence of blanks, newlines, and tabs
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
CS 143 Lecture 3 6 What are Tokens For? Classify program substrings according to role Output of lexical analysis is a stream of tokens . . . . . . which is input to the parser Parser relies on token distinctions An identifier is treated differently than a keyword
Background image of page 6
CS 143 Lecture 3 7 Designing a Lexical Analyzer: Step 1 Define a finite set of tokens Tokens describe all items of interest Choice of tokens depends on language, design of parser
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
CS 143 Lecture 3 8 Example Recall \tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1; Useful tokens for this expression: Integer, Keyword, Relation, Identifier, Whitespace, (, ), =, ; Note: ( , ), = , ; are tokens, not characters, here
Background image of page 8
CS 143 Lecture 3 9 Designing a Lexical Analyzer: Step 2 Describe which strings belong to each token Recall: Identifier: strings of letters or digits, starting with a letter
Background image of page 9

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 10
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}