This preview shows page 1. Sign up to view the full content.
Unformatted text preview: CS 124/LINGUIST 180: From
Language to Information
Dan Jurafsky Lecture 3: Intro to Probability, Language Modeling IP notice: some slides for today from: Jim Martin, Sandiway Fong, Dan Klein Outline
Probability Basic probability Condi?onal probability Language Modeling (N
grams) N
gram Intro The Chain Rule The Shannon Visualiza?on Method Evalua?on: Perplexity Smoothing: Laplace (Add
1) Add
prior 1. Introduction to Probability
Experiment (trial) Repeatable procedure with well
deﬁned possible outcomes Sample Space (S) the set of all possible outcomes ﬁnite or inﬁnite Example coin toss experiment possible outcomes: S = {heads, tails} Example die toss experiment possible outcomes: S = {1,2,3,4,5,6} Slides from Sandiway Fong Introduction to Probability
Deﬁni?on of sample space depends on what we are asking Sample Space (S): the set of all possible outcomes Example die toss experiment for whether the number is even or odd possible outcomes: {even,odd} not {1,2,3,4,5,6} More definitions
Events an event is any subset of outcomes from the sample space Example die toss experiment let A represent the event such that the outcome of the die toss experiment is divisible by 3 A = {3,6} A is a subset of the sample space S= {1,2,3,4,5,6} Example Draw a card from a deck suppose sample space S = {heart,spade,club,diamond} (four suits) let A represent the event of drawing a heart let B represent the event of drawing a red card A = {heart} B = {heart,diamond} Introduction to Probability
Some deﬁni?ons Coun?ng suppose opera?on oi can be performed in ni ways, then a sequence of k opera?ons o1o2...ok can be performed in n1 × n2 × ... × nk ways Example die toss experiment, 6 possible outcomes two dice are thrown at the same ?me number of sample points in sample space = 6 × 6 = 36 Definition of Probability
The probability law assigns to an event a nonnega?ve number Called P(A) Also called the probability A That encodes our knowledge or belief about the collec?ve likelihood of all the elements of A Probability law must sa?sfy certain proper?es Probability Axioms
Nonnega?vity P(A) >= 0, for every event A Addi?vity If A and B are two disjoint events, then the probability of their union sa?sﬁes: P(A U B) = P(A) + P(B) Normaliza?on The probability of the en?re sample space S is equal to 1, I.e. P(S) = 1. An example
An experiment involving a single coin toss There are two possible outcomes, H and T Sample space S is {H,T} If coin is fair, should assign equal probabili?es to 2
outcomes Since they have to sum to 1 P({H}) = 0.5 P({T}) = 0.5 P({H,T}) = P({H})+P({T}) = 1.0 Another example
Experiment involving 3 coin tosses Outcome is a 3
long string of H or T S ={HHH,HHT,HTH,HTT,THH,THT,TTH,TTT} Assume each outcome is equiprobable “Uniform distribu?on” What is probability of the event that exactly 2 heads
occur? A = {HHT,HTH,THH} P(A) = P({HHT})+P({HTH})+P({THH}) = 1/8 + 1/8 + 1/8 =3/8 Probability definitions
In summary: Probability of drawing a spade from 52 well
shuﬄed playing cards: Probabilities of two events
If two events A and B are independent Then P(A and B) = P(A) x P(B) If ﬂip a fair coin twice What is the probability that they are both heads? If draw a card from a deck, then put it back, draw a card from the deck again What is the probability that both drawn cards are hearts? How about nonuniform
probabilities? An example
A biased coin, twice as likely to come up tails as heads, is tossed twice What is the probability that at least one head occurs? Sample space = {hh, ht, th, j} (h = heads, t = tails) Sample points/probability for the event: ht 1/3 x 2/3 = 2/9
th 2/3 x 1/3 = 2/9
hh 1/3 x 1/3= 1/9
j 2/3 x 2/3 = 4/9 Answer: 5/9 = ≈0.56 (sum of weights in red) Moving toward language
What’s the probability of drawing a 2 from a deck of 52 cards with four 2s? 4
1
P ( drawing a two) =
= = .077
52 13
What’s the probability of a random word (from a random dic?onary page) being a verb? € € # of ways to get a verb
P ( drawing a verb) =
all words Probability and part of speech tags
What’s the probability of a random word (from a random dic?onary page) being a verb? P ( drawing a verb) = # of ways to get a verb
all words How to compute each of these € All words = just count all the words in the dic?onary # of ways to get a verb: number of words which are verbs! If a dic?onary has 50,000 entries, and 10,000 are verbs…. P (V) is 10000/50000 = 1/5 = .20 Conditional Probability
A way to reason about the outcome of an experiment based on par?al informa?on In a word guessing game the ﬁrst lejer for the word is a “t”. What is the likelihood that the second lejer is an “h”? How likely is it that a person has a disease given that a medical test was nega?ve? A spot shows up on a radar screen. How likely is it that it corresponds to an aircrap? More precisely
Given an experiment, a corresponding sample space S, and a probability law Suppose we know that the outcome is within some given event B We want to quan?fy the likelihood that the outcome also belongs to some other given event A. We need a new probability law that gives us the condi?onal probability of A given B P(AB) An intuition
A is “it’s raining now”. P(A) in dry California is .01 B is “it was raining ten minutes ago” P(AB) means “what is the probability of it raining now if it was raining 10 minutes ago” P(AB) is probably way higher than P(A) Perhaps P(AB) is .10 Intui?on: The knowledge about B should change our es?mate of the probability of A. Conditional probability
One of the following 30 items is chosen at random What is P(X), the probability that it is an X? What is P(Xred), the probability that it is an X given that it is red? Conditional Probability
let A and B be events p(BA) = the probability of event B occurring given event A occurs deﬁni8on: p(BA) = p(A ∩ B) / p(A) S Conditional probability
P(AB) = P(A ∩ B)/P(B) Or P( A  B) = P( A, B)
P( B) Note: P(A,B)=P(AB) · P(B)
Also: P(A,B) = P(B,A)
A A,B B Independence
What is P(A,B) if A and B are independent? P(A,B)=P(A) P(B) iﬀ A,B independent. P(heads,tails) = P(heads) P(tails) = .5 .5 = .25
Note: P(AB)=P(A) iﬀ A,B independent Also: P(BA)=P(B) iﬀ A,B independent Summary
Probability Condi?onal Probability Independence Language Modeling
We want to compute P(w1,w2,w3,w4,w5…wn) = P(W) = the probability of a sequence Alterna?vely we want to compute P(w5w1,w2,w3,w4) =the probability of a word given some previous words The model that computes P(W) or P(wnw1,w2…wn
1) is called the language model. A bejer term for this would be “ The Grammar” But “Language model” or LM is standard Computing P(W)
How to compute this joint probability: P(“the” , ”other” , ”day” , ”I” , ”was” , ”walking” , ”along”, ”and”,”saw”,”a”,”lizard”) Intui?on: let’s rely on the Chain Rule of Probability The Chain Rule
Recall the deﬁni?on of condi?onal probabili?es P( A^ B)
P( A  B) =
P( B) Rewri?ng: P( A^ B) = P( A  B) P( B)
More generally P(A,B,C,D) = P(A)P(BA)P(CA,B)P(DA,B,C) In general P(x1,x2,x3,…xn) = P(x1)P(x2x1)P(x3x1,x2)…P(xnx1…xn
1) The Chain Rule applied to joint
probability of words in sentence P(“the big red dog was”)= P(the) * P(bigthe) * P(redthe big) * P(dogthe big red) * P(wasthe big red dog) Very easy estimate:
How to es?mate? P(the  its water is so transparent that) P(the  its water is so transparent that) = C(its water is so transparent that the) ___________________________________________________________________________________ C(its water is so transparent that) Unfortunately
There are a lot of possible sentences We’ll never be able to get enough data to compute the sta?s?cs for those long preﬁxes P(lizard the,other,day,I,was,walking,along,and,saw,a) Or P(theits water is so transparent that) Markov Assumption
Make the simplifying assump?on P(lizardthe,other,day,I,was,walking,along,and,saw,a) = P (lizarda) Or maybe P(lizardthe,other,day,I,was,walking,along,and,saw,a) = P (lizardsaw,a) Markov Assumption
" So for each component in the product replace with
the approximation (assuming a prefix of N) n −1
1 P ( wn  w ) ≈ P ( wn  w n −1
n − N +1 " Bigram version n −1
1 P(wn  w ) ≈ P ( w n  w n −1 ) ) Estimating bigram probabilities
The Maximum Likelihood Es?mate count ( w i−1, w i )
P ( w i  w i−1 ) =
count ( w i−1 )
c ( w i−1, w i )
P ( w i  w i−1 ) =
c ( w i−1 ) An example
<s> I am Sam </s> <s> Sam I am </s> <s> I do not like green eggs and ham </s> This is the Maximum Likelihood Es?mate, because it is the one which maximizes P(Training setModel) Maximum Likelihood Estimates
The maximum likelihood es?mate of some parameter of a model M from a training set T Is the es?mate that maximizes the likelihood of the training set T given the model M Suppose the word Chinese occurs 400 ?mes in a corpus of a million words (Brown corpus) What is the probability that a random word from some other text will be “Chinese” MLE es?mate is 400/1000000 = .004 This may be a bad es?mate for some other corpus But it is the es'mate that makes it most likely that “Chinese” will occur 400 ?mes in a million word corpus. More examples: Berkeley
Restaurant Project sentences
can you tell me about any good cantonese restaurants close by mid priced thai food is what i’m looking for tell me about chez panisse can you give me a lis?ng of the kinds of food that are available i’m looking for a good place to eat breakfast when is caﬀe venezia open during the day Raw bigram counts
Out of 9222 sentences Raw bigram probabilities
Normalize by unigrams: Result: Bigram estimates of sentence
probabilities
P(<s> I want english food </s>) = P(i<s>) x P(wantI) x P(englishwant) x P(foodenglish) x P(</s>food) =.000031 What kinds of knowledge?
P(englishwant) = .0011 P(chinesewant) = .0065 P(towant) = .66 P(eat  to) = .28 P(food  to) = 0 P(want  spend) = 0 P (i  <s>) = .25 The Shannon Visualization Method
Generate random sentences: Choose a random bigram <s>, w according to its probability Now choose a random bigram (w, x) according to its probability And so on un?l we choose </s> Then string the words together <s> I I want want to to eat eat Chinese Chinese food food </s> Approximating Shakespeare
Shakespeare as corpus
N=884,647 tokens, V=29,066 Shakespeare produced 300,000 bigram types out of V2= 844 million possible bigrams: so, 99.96% of the possible bigrams were never seen (have zero entries in the table) Quadrigrams worse: What's coming out looks like Shakespeare because it is Shakespeare The wall street journal is not
shakespeare (no offense)
Lesson 1: the perils of overfitting
N
grams only work well for word predic?on if the test corpus looks like the training corpus In real life, it open doesn’t We need to train robust models, adapt to test set, etc Lesson 2: zeros or not?
Zipf’s Law: A small number of events occur with high frequency A large number of events occur with low frequency You can quickly collect sta?s?cs on the high frequency events You might have to wait an arbitrarily long ?me to get valid sta?s?cs on low frequency events Result: Our es?mates are sparse! no counts at all for the vast bulk of things we want to es?mate! Some of the zeroes in the table are really zeros But others are simply low frequency events you haven't seen yet. Aper all, ANYTHING CAN HAPPEN! How to address? Answer: Es?mate the likelihood of unseen N
grams! Slide adapted from Bonnie Dorr and Julia Hirschberg Smoothing is like Robin Hood:
Steal from the rich and give to the poor (in
probability mass) Slide from Dan Klein Laplace smoothing
Also called add
one smoothing Just add one to all the counts! Very simple MLE es?mate: Laplace es?mate: Reconstructed counts: Laplace smoothed bigram counts Laplacesmoothed bigrams Reconstituted counts Note big change to counts
C(count to) went from 608 to 238! P(towant) from .66 to .26! Discount d= c*/c d for “chinese food” =.10!!! A 10x reduc?on So in general, Laplace is a blunt instrument But Laplace smoothing not used for N
grams, as we have much bejer methods Despite its ﬂaws Laplace (add
k) is however s?ll used to smooth other probabilis?c models in NLP, especially For pilot studies in domains where the number of zeros isn’t so huge. Addk
Add a small frac?on instead of 1 Even better: Bayesian unigram
prior smoothing for bigrams
Maximum Likelihood Es?ma?on C ( w1, w 2 )
P ( w 2  w1 ) =
C ( w1 )
Laplace Smoothing € C ( w1, w 2 ) + 1
PLaplace ( w 2  w1 ) =
C ( w1 ) + vocab Bayesian Prior Smoothing € C ( w1, w 2 ) + P ( w 2 )
PPrior ( w 2  w1 ) =
C ( w1 ) + 1 Practical Issues
We do everything in log space Avoid underﬂow (also adding is faster than mul?plying) Language Modeling Toolkits
SRILM hjp://www.speech.sri.com/projects/ srilm/ Google NGram Release Google NGram Release
serve as the incoming 92!
serve as the incubator 99!
serve as the independent 794!
serve as the index 223!
serve as the indication 72!
serve as the indicator 120!
serve as the indicators 45!
serve as the indispensable 111!
serve as the indispensible 40!
serve as the individual 234! http://googleresearch.blogspot.com/2006/08/allourngramarebelongtoyou.html Evaluation
We train parameters of our model on a training set. How do we evaluate how well our model works? We look at the models performance on some new data This is what happens in the real world; we want to know how our model performs on data we haven’t seen So a test set. A dataset which is diﬀerent than our training set Then we need an evalua'on metric to tell us how well our model is doing on the test set. One such metric is perplexity (to be introduced below) Evaluating Ngram models
Best evalua?on for an N
gram Put model A in a task (language iden?ﬁca?on, speech recognizer, machine transla?on system) Run the task, get an accuracy for A (how many lgs iden?ﬁed corrrectly, or Word Error Rate, or etc) Put model B in task, get accuracy for B Compare accuracy for A and B Extrinsic evalua'on Difficulty of extrinsic (invivo)
evaluation of Ngram models
Extrinsic evalua?on This is really ?me
consuming Can take days to run an experiment So As a temporary solu?on, in order to run experiments To evaluate N
grams we open use an intrinsic evalua?on, an approxima?on called perplexity But perplexity is a poor approxima?on unless the test data looks just like the training data So is generally only useful in pilot experiments (generally is not suﬃcient to publish) But is helpful to think about. Perplexity
Perplexity is the probability of the test set (assigned by the language model), normalized by the number of words: Chain rule: For bigrams: " Minimizing perplexity is the same as maximizing probability
" The best language model is one that best predicts an
unseen test set A totally different perplexity
Intuition
How hard is the task of recognizing digits ‘0,1,2,3,4,5,6,7,8,9,oh’: easy, perplexity 11 (or if we ignore ‘oh’, perplexity 10) How hard is recognizing (30,000) names at Microsop. Hard: perplexity = 30,000 If a system has to recognize Operator (1 in 4) Sales (1 in 4) Technical Support (1 in 4) 30,000 names (1 in 120,000 each) Perplexity is 54 Perplexity is weighted equivalent branching factor Slide from Josh Goodman Perplexity as branching factor Lower perplexity = better model
Training 38 million words, test 1.5 million words, WSJ Unknown words: Open versus
closed vocabulary tasks
If we know all the words in advanced Vocabulary V is ﬁxed Closed vocabulary task OHen we don’t know this Out Of Vocabulary = OOV words Open vocabulary task Instead: create an unknown word token <UNK> Training of <UNK> probabili?es Create a ﬁxed lexicon L of size V At text normaliza?on phase, any training word not in L changed to <UNK> Now we train its probabili?es like a normal word At decoding ?me If text input: Use UNK probabili?es for any word not in training Advanced LM stuff
Current best smoothing algorithm Kneser
Ney smoothing Other stuﬀ Interpola?on Backoﬀ Variable
length n
grams Class
based n
grams Clustering Hand
built classes Cache LMs Topic
based LMs Sentence mixture models Skipping LMs Parser
based LMs Backoff and Interpolation
Another really useful source of knowledge If we are es?ma?ng: trigram p(zxy) but c(xyz) is zero Use info from: Bigram p(zy) Or even: Unigram p(z) How to combine the trigram/bigram/unigram info? Backoff versus interpolation
Backoﬀ: use trigram if you have it, otherwise bigram, otherwise unigram Interpola'on: mix all three Interpolation
Simple interpola?on Lambdas condi?onal on context: How to set the lambdas?
Use a held
out corpus Training Data HeldOut
Data Test
Data Choose lambdas which maximize the probability of thisheld
out data I.e. ﬁx the N
gram probabili?es Then search for lambda values That when plugged into previous equa?on Give largest probability for held
out set Can use EM to do this search Intuition of backoff+discounting
How much probability to assign to all the zero trigrams? Use GT or other discoun?ng algorithm to tell us How to divide that probability mass among diﬀerent contexts? Use the N
1 gram es?mates to tell us What do we do for the unigram words not seen in training? Out Of Vocabulary = OOV words ARPA format Summary
Probability Basic probability Condi?onal probability Language Modeling (N
grams) N
gram Intro The Chain Rule The Shannon Visualiza?on Method Evalua?on: Perplexity Smoothing: Laplace (Add
1) Add
k Add
prior ...
View
Full
Document
This document was uploaded on 06/01/2011.
 Winter '09

Click to edit the document details