Lecture-5.pptx - Text Preprocessing Techniques By: Dr. Ola...

This preview shows page 1 - 10 out of 33 pages.

Text PreprocessingTechniquesBy: Dr. Ola SurakhiEmail: [email protected]
IntroductionNLP ToolkitsThe most popular NLP libraryPython libraries for natural language processingText Preprocessing TechniquesConverting text to a meaningful format for analysisPreprocessing and cleaning text
NLP ToolkitsCode: How to Install NLTKpip install nltkimport nltknltk.download()# downloads all data & models# this will take a while
Sample Text DataHi Mr. Smith! I’m going to buy some vegetables (tomatoes andcucumbers) from the store. Should I pick up some black-eyed peasas well?To analyze this data, we need to preprocess and normalize the text
Preprocessing Techniques:TokenizationTokenization = splitting raw text into small, indivisible units forprocessingThese units can be:WordsSentencesN-gramsOther characters defined by regular expressions
Tokenization: Space-basedtokenizationA very simple way to tokenizeFor languages that use space characters between wordsArabic, Cyrillic, Greek, Latin, etc., based writing systemsSegment off a token between instances of spaces
Simple Tokenization in Python(Words)
Issues in TokenizationCan't just blindly remove punctuation:m.p.h., Ph.D., AT&T, cap’nprices($45.55)dates (01/02/06)URLs ()hashtags (#nlproc)email addresses ([email protected])Clitic: a word that doesn't stand on its own"are"inwe're,French "je" inj'ai,"le" inl'honneurWhen should multiword expressions (MWE) be words?New York, rock ’n’ roll
Tokenization in languages withoutspacesMany languages (like Chinese, Japanese, Thai) don't use spaces toseparate words!

Upload your study docs or become a

Course Hero member to access this document

Upload your study docs or become a

Course Hero member to access this document

End of preview. Want to read all 33 pages?

Upload your study docs or become a

Course Hero member to access this document

Term
Spring
Professor
abdallah
Tags
Word processor, Yao Ming, Morpheme, Lexeme

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture