TokenizationAndParsing - Tokenization and parsing We have...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Tokenization and parsing We have implemented a small calculator. We can enter an expression, the calculator prints the value of the expression. How does the calculator get from the expression (which is just a string) to the value? This process is a very simple version of what happens in interpreters or compilers for programming languages (e.g. the Javascript interpreter built into your web browser, the macro languages of Excel, Maple, or Matlab, or even the Java compiler itself). The process consists of three steps: lexical analysis , syntactic analysis and semantic analysis . Lexical analysis Lexical analysis ( tokenization ) splits the input text (a string or a text file) into minimal meaningful units. The minimal units are called tokens , and lexical analyzers are also called tokenizers . For our calculator we need three kinds of tokens: • numbers (real numbers, such as 3, 71, 23 . 45), • identifiers (variable names, like i , j , a123 ), • operators ( + ,- , * , / , ··· ) If the expression below is given to a tokenizer (abc12+27 * 23.0(12abc34 the result would be the following: symbol ( identifier abc12 symbol + number 27.0 symbol * number 23.0 symbol ( number 12.0 identifer abc34 The tokenizer does not know anything about the meaning or syntax of expressions. Even though the string above is nonsense (or at least it is not a meaningful expression), the tokenizer has no difficulty to split it...
View Full Document

This note was uploaded on 04/20/2010 for the course CS CS206 taught by Professor Lee during the Fall '09 term at 카이스트, 한국과학기술원.

Page1 / 4

TokenizationAndParsing - Tokenization and parsing We have...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online