02._Lexical_Analysis

02._Lexical_Analysis - Lexical Analysis Teodor Przymusinski...

Info icon This preview shows pages 1–7. Sign up to view the full content.

View Full Document Right Arrow Icon
Lexical Analysis Teodor Przymusinski, UC Riverside Compiler Construction CS152 - Based on the slides by Leonidas Fegaras, UTA
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Lexical Analysis A scanner groups input characters into tokens input token value keyword let identifier x equal = identifier x star * let x = x *(a+123) left-paren ( identifier a plus + integer 123 right-paren ) Tokens are typically represented by numbers 2 Compiler Construction CS152 - Based on the slides by Leonidas Fegaras, UTA
Image of page 2
Parser Each time the parser needs a token, it sends a request to the scanner the scanner reads as many characters from the input stream as necessary to construct a single token when a single token is formed, the scanner is suspended and returns the token to the parser the parser will repeatedly call the scanner to read all the tokens from the input stream scanner parser get token token source file get next character AST 3 Compiler Construction CS152 - Based on the slides by Leonidas Fegaras, UTA
Image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Tasks of a Scanner A typical scanner: recognizes the keywords of the language these are the reserved words that have a special meaning in the language, such as the word class in Java recognizes special characters, such as ( and ), or groups of special characters, such as := and == recognizes identifiers, integers, reals, decimals, strings, etc ignores whitespaces (tabs, blanks, etc) and comments recognizes and processes special directives (such as the #include "file" directive in C) and macros 4 Compiler Construction CS152 - Based on the slides by Leonidas Fegaras, UTA
Image of page 4
Scanner Generators Input: a scanner specification describes every token using Regular Expressions (REs) e.g., the RE [a-z][a-zA-Z0-9]* recognizes all identifiers with at least one alphanumeric letter whose first letter is lower-case alphabetic handles whitespaces and resolve ambiguities Output: the actual scanner Scanner generators compile regular expressions into efficient programs (finite state machines) You will use a scanner generator for Java, called JLex, for the project 5 Compiler Construction CS152 - Based on the slides by Leonidas Fegaras, UTA
Image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Regular Expressions are a very convenient form of representing (possibly infinite) sets of strings, called regular sets e.g., the RE (a | b)*aa represents the infinite set {“aa”,“aaa”,“baa”,“abaa”, ... } a RE is one of the following: Name RE Designation epsilon ε {“”} symbol a {“a”} for some character a concatenation AB the set { rs | r A , s }, where rs is a concatenation, and A and B designate the REs from A and B alternation A | B the set A     B , where A and B designate the REs from A and B repetition A* the set ε | A | (AA) | (AAA) | ... (an infinite set) e.g., the RE (a | b)c designates { rs | r {“a”} {“b”}, s  {“c”} }, which is equal to {“ac”,“bc”} Shortcuts: P +  = PP*,  P? = P |  ε ,   [a-z] = (“a”|“b”|...|“z”) 6 Compiler Construction CS152 - Based on the slides by Leonidas Fegaras, UTA
Image of page 6
Image of page 7
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern