tokenizingmachine

tokenizingmachine - //

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: // /*----------------------------------------------------------------------*\ // | Concrete Template Body : Tokenizing_Machine_Kernel_1_Body // \*----------------------------------------------------------------------*/ #ifndef CT_TOKENIZING_MACHINE_KERNEL_1_BODY #define CT_TOKENIZING_MACHINE_KERNEL_1_BODY 1 ///------------------------------------------------------------------------ /// Global Context -------------------------------------------------------- ///------------------------------------------------------------------------ #include "CT/Tokenizing_Machine/Kernel_1.h" ///------------------------------------------------------------------------ /// Public Operations ----------------------------------------------------- ///------------------------------------------------------------------------ concrete_template < concrete_instance class Rep > procedure_body Tokenizing_Machine_Kernel_1 < Rep > :: Insert ( preserves Character ch ) { select (self[buffer_type]) { case EMPTY_STRING : // entire content of buffer will be ch { Set_Buffer_And_Buffer_Type (ch, self[buffer], self[buffer_type]); } break; case QUESTION_MARK : { if (Is_Id_Var_Character (ch)) // building a variable token { self[buffer].Add (self[buffer].Length (), ch); self[buffer_type] = VARIABLE; } else if (Can_Start_Token (ch)) // buffer holds error token, restarting new token { Prepare_To_Dispense (ERROR, self[buffer], self[buffer_type], self[text_for_token], self[kind_of_token], self[token_ready] ); Set_Buffer_And_Buffer_Type (ch, self[buffer], self[buffer_type]); } else // continuing to build an error token { self[buffer].Add (self[buffer].Length (), ch); self[buffer_type] = ERROR; } } break; case VARIABLE : { if (Is_Id_Var_Character (ch)) // continuing to build a variable token { self[buffer].Add (self[buffer].Length (), ch); } else // buffer holds complete variable token { Prepare_To_Dispense (VARIABLE, self[buffer], self[buffer_type], self[text_for_token], self[kind_of_token], self[token_ready]); Set_Buffer_And_Buffer_Type (ch, self[buffer], self[buffer_type]); } } break; case ID_OR_OPRTR : { if (Is_Id_Var_Character (ch)) // continuing to build identifier or operator token { self[buffer].Add (self[buffer].Length (), ch); } else // buffer holds complete identifier or operator token { object Integer token_kind; token_kind = Id_Or_Operator (self[buffer]); Prepare_To_Dispense (token_kind, self[buffer], self[buffer_type], self[text_for_token], self[kind_of_token], self[token_ready]); Set_Buffer_And_Buffer_Type (ch, self[buffer], self[buffer_type]); } } break;...
View Full Document

This note was uploaded on 02/27/2012 for the course CSE 321 taught by Professor Bucci during the Winter '11 term at Ohio State.

Page1 / 10

tokenizingmachine - //

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online