Institute Massachusetts of Technology Department of Electrical Engineering & Computer Science 6.345/HST.728 Automatic Speech Recognition Spring, 2010 4/13/10
Lecture Handouts
VQ-based HMMs Discriminative Training
4/13/10
Addendum on VQ-based systems
So far, have modeled the output distribution of a frame y as a mixture distribution Another approach is based on vector quantization (codebooks)
Replace y by a label, drawn from some finite set Codebooks are generated via clustering Now, the output distribution of a state is a multinomial distribution over these vq labels
Let y* be the vq label that replaces the observation y Then the probability of observing vq label v in state i is P(y* = v | x = i) = fi (v), where v = 1,...,C C is the number of vq labels, and we constrain fi (v) = 1
v
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 79
Advantages and Disadvantages of VQ
Advantages
No parametric assumptions about the shape of the output distribution Computationally fast: * Obtain a likelihood by one table look-up
Disadvantages
Lose resolution in the data representation * Must carve up a 50 dimensional space into a few hundred regions * But can improve matters Multiple codebooks, one for each subset of features This improves resolution but at the expense of introducing further independence assumptions
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 80
1
4/13/10
Estimating output distributions for VQ systems
We run BW as usual to obtain a probabilistic alignment. We compute t (i ), the probability of frame t being assigned to state i Let I v (Y * ) = 1 if Y * = v 0 if Y * v
so I v is an indicator function for the event that Y * = v Now to estimate the probability of observing label v in state i, we compute
(i) I (Y
t v
*
t
)
fi (v) =
t
(i)
t t
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 81
An Introduction to Discriminative Training
Larry Gillick
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 82
2
4/13/10
Origin of the Idea
There are many estimation methods Maximum likelihood estimation is asymptotically optimal in most situations
Originally conjectured by the great statistician Ronald Fisher in 1925 Theres been a huge body of theoretical work since then
So why should you use any other method?
What if the model is wrong!
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 83
A simple example
Consider a simple classification problem Observe Y and decide which of two equally likely hypotheses is true H 0 :Y is drawn from class C 0 H1 :Y is drawn from class C1 Assume that if Y is from C 0 , then Y ~ N(0,1) if Y is from C1 , then Y ~ N(,1) Assume the two error types have equal costs Standard Bayesian reasoning implies the best decision rule is Choose H1 if Y > / 2 Choose H 0 otherwise Since is unknown, must estimate it; use maximum likelihood
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 84
3
4/13/10
Not so simple?
Suppose model for C 0 is correct but model for C1 is not So Y ~ N(0,1) if Y is from C 0 but Y ~ N(0,100 2 ) if Y is from C1 Let' s, nonetheless, proceed to estimate using ML and assume we have a (virtually) infinite supply of data Our estimate for will be (very near) 0, and our decision rule will be choose H 1 if Y > 0, otherwise choose H 0 Our error rate will be around 50%!
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 85
Is this the best we can do?
Focus on the decision rule instead of the estimation procedure! Consider the decision rule : choose H 1 if Y > k What would be a good choice for k? How about k = 5? If Y is from class C 0 , then we will almost always correctly choose H 0 If Y is from class C1 , then we will choose H1 almost 50% of the time If the prior probabilites are both .5, then our error rate will be around 25% So there exists a better value for k than the one obtained via the MLE!
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 86
4
4/13/10
The Two-class problem again
Consider again the two class problem H 0 :Y is from class C 0 with dist Y ~ N( 0 , ) H1 :Y is from class C1 with dist Y ~ N(1 , ) Assume prior probabilities p 0 and p1 , which sum to 1 Let f0 (y) and f1 (y) be the prob densities for Y under H 0 and H 1 The Bayes classification rule is to decide H1 if the likelihood ratio exceeds a constant k P(H1 | Y = y) p1 f1 (y) = >k P(H 0 | Y = y) p0 f0 (y) where k is chosen based on the relative costs of the two types of errors
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 87
We do some algebra...
Observe that log fi (y) = C - (y - i )-1 (y - i ) / 2 where C does not depend on i (it is the same for both dists) log( p1 f1 (y) / p0 f0 (y)) = 0 + (1 - 0 )-1 y where 0 is a constant that does not depend on y, and we have relied on the symmetry of So the Bayes classifier is of the form : Decide H1 if 0 + y > k where = -1 (1 - 0 ) the classic " linear discriminant" formula
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 88
5
4/13/10
The discriminative strategy
We can then substitute the standard MLEs for p0 , p1 , 0 , 1 , to obtain the desired decision rule. But suppose we want to assume the form of this Bayes rule decide H1 if 0 + y > k But we do not wish to make the standard normal assumptions and then estimate 0 , 1 , and via ML Instead, we estimate the classification formula directly!
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 89
Logistic Regression
Let X be the true class of the data, so X = 0 or X = 1 p1 f1 (y) 1 1 1 1. P(X = 1 | Y = y) = = = = p1 f1 (y ) -(0 +y ) p0 f0 (y) - log( p0 f0 (y) + p1 f1 (y) 1+ 1+ e 1+ e p0 f0 (y ) p1 f1 (y) Now ignore the formula we derived for in terms of the Gaussian means and covariance matrix Instead, assume we observe a sequence of pairs : (X1 ,Y1 ),...,(X n ,Yn ) and we seek estimates for the constant term 0 , and the regression coefficients using the logistic regression formula 1.
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 90
6
4/13/10
More specifically...
We regard Y as fixed (we condition on Y) and we regard X as variable. So we write down the conditional likelihood. 1 e i 1-xi L(X | Y , ) = P (X = xi | Y = yi ) = ( ) xi ( ) 1+ e- yi 1+ e- yi i=1 i=1
For simplicity, we have amalgamated the constant term 0 into the vector of unknown parameters to be estimated, by adding an additional dimension to the Y vector and putting a 1 there.
n
n
- y
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 91
Conditional Maximum Likelihood Estimation
We then maximize the conditional likelihood over CMLE = arg max L(X | Y , )
The obtained in this way will not be the same as what we would obtain from the use of the unconditional MLEs obtained via the generative model. If the generative model were correct, then the CMLE would be inferior. However, if the generative model is incorrect, then the CMLE will generally be superior. See Efron(1975) " The Efficiency of Logistic Regression..."
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 92
7
4/13/10
Comparison of Generative vs Discriminative training MLE vs CMLE Y is regarded as random vs X is regarded as random Generative vs Disciminative
Estimation via how Y is generated Estimation via the classifier
Discriminative method is more robust to departures of the data from the model
Or, at least, that is the hope!
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 93
Kullback-Leibler divergence
Kullback - Leibler divergence between two distributions is f (x) DKL ( f , g) = f (x) log( )dx g(x) the expected value of the log likelihood ratio of the true distribution f(x) to some other distribution g(x) Measures how much bigger the log likelihood of the true dist is than that of some impostor dist, on the average. You can use Jensen s inequality to prove that DKL ( f , g) > 0 if f g
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 94
8
4/13/10
Mutual Information
Mutual Information of two random variables W,Y with joint density f(w,y) and marginal densities g(w) and h(y) f (w, y) I(W,Y ) = KL( f , gh) = f (w, y) log( )dwdy g(w)h(y) f (W,Y ) f (W | Y )h(Y ) = E f [log( )] = E f [log( )] g(W )h(Y ) g(W )h(Y ) f (W | Y ) = E f [log( )] = E f [log( f (W | Y ))] - E g [log(g(W ))] g(W ) = H (W ) - H (W | Y ) where H(W) is the entropy of W and H(W | Y) is the conditional entropy; so I(W, Y) is the reduction in the entropy of W given Y
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 95
Maximum Mutual Information training (MMI)
MLE approach to training is to choose to maximize
M
1.
P (y | w )
i i i =1 M
or, equivalently, to maximize 2. log(P (yi | wi ))
i=1
MMI approach to training is to choose to maximize
M
3. P (wi | yi )
i=1
or, equivalently, to maximize
M
4. log(P (wi | yi ))
i=1
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 96
9
4/13/10
Comments on MMI
Note the parallel to
logistic regression vs linear discriminant classifier
MMI training is really just CMLE
Conditional maximum likelihood estimation Regard the transcription as the random variable and frame the sequence as fixed
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 97
Correcting for unmodeled dependence
Ideally, TotScr=AcModScr+LangModScr But, actually must use shrinkage factor 1/k
TotScr=(1/k)*AcModScr+LangModScr To compensate for dependence
Why? Imagine repeating each frame
Thereby doubling the AcModScr To compensate, youd need to set k=2 * Effectively taking the square root of P(Y|W)
In practice, k is "tuned" for accuracy
Very approximate compensation for both frame and feature dependence
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 98
10
4/13/10
LM role in MMI training
In MMI training, we seek to find to maximize
P (w | y ) over a large set of training utts y .
i i i i
But
P (wi | yi ) =
P (yi | wi )1/k P(wi ) P (yi )
where P(wi ) refers to the language model probability of the word string wi - - so just as with recognition, we must de - weight the effect of the acoustic model during training. Often, in MMI training it can be tricky to decide what LM to use because the " task" at recognition time might be quite different from the task represented by the training data!
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 99
Normalization
Recall the normalization factor in P(w|y)
P (y) = P (y | w )1/k P( w )
where w ranges over all possible transcriptions of the acoustics y.
For the MLE, it is irrelevant but for MMI, it is at the heart of the computation Cant sum over all transcriptions, so approximate with
Nbest list Lattice
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 100
11
4/13/10
Generating alternative hypotheses
Instead of generating just the best hypothesis corresponding to an utterance -- you can generate the n best hypotheses
We wont describe the necessary algorithmic modifications but theres a large research literature on how to do this efficiently We can record these hypotheses (and score information) in a long list (an nbest list) But can store this information more compactly in a lattice * These hypotheses can be thought of as "competitors" of the correct hypothesis * But note that the correct hypothesis should also be in the lattice * A future lecture will deal with lattices
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 101
How to compute MMI estimates
EM algorithm doesnt help much when computing the MMI (or CMLE) estimate
Too hard to compute the expected value of the conditional loglikelihood: remember Q?
Initially, the computational approach was via gradient descent Now, theres a modification of BW: extended BW
Justified heuristically Not so hard to compute from lattices
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 102
12
4/13/10
Baum-Welch again
Recall Baum-Welch training
Each frame is assigned to a set of possible states: "Fractional frames" Estimate a Gaussian from all fractional frames assigned to it So, for example, the MLE for the mean of a certain state is
MLE
Y
=
t
t t
t
,
t
where t is the fraction of thet' th frame assigned to the state, and we assume a single Gaussian for its output distribution
Here we can assume that we have run multiple iterations, and estimates have converged
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 103
Extended Baum-Welch (1)
Assume we already have the MLE estimates In MMI training, there are two alignment processes
One for the correct hypothesis * The numerator statistics P(y|w)P(w) One for the competitor hypotheses represented by nbest list or lattice * The denominator statistics P(y)
Accumulate fractional frame statistics for both alignments
So, conceptually, for each hypothesis in the nbest list, we blow apart each frame into fractional frames assigned to states
Then we compute the next estimate of the mean by combining the numerator and denominator statistics This processs can be iterated
Typically iterated without recomputing the lattice of alternatives
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 104
13
4/13/10
Extended Baum-Welch (2)
Let y (num) = the weighted avg of fractional frames assigned to the Gaussian using the correct hypothesis Let C (num) = the count (sum) of the corresponding fractional frame probabilities By analogy, Let y (denom) = the weighted avg of fractional frames assigned to the Gaussian using all of the alternative hypotheses in the denominator lattice Let C (denom) = the count (sum) of the corresponding fractional frame probabilities Then, the resulting estimate on the next iteration of will be C (num)y (num) - C (denom)y (denom) C (num ) -C (denom)
=
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 105
Interpretation of Extended BW
We are, in a sense, trying to move the mean away from trouble The negative term represents an accumulation of frames that are mistakenly assigned to the state The positive term represents an accumulation of frames that are correctly assigned to the state Think of 2 points in space, the mean of the good data and the mean of the bad data Extended BW moves the mean of the good data away from the mean of the bad data along the line connecting them We can rewrite the formula more compactly as = wg yg - wb yb
= yg +[(wg -1)yg - wb yb ]
where g and b denote the statistics for the "good" and " bad" examples So we add a vector to yg pointing roughly from yb to yg , moving the estimate away from the mean of " impostor" frames
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 106
14
4/13/10
Extended BW with smoothing
There is nothing to prevent negative counts from swamping the positive counts, so we must smooth We smooth by taking a weighted average with our prior estimate The real formula is this:
C (num)y (num) - C (denom)y (denom) + D C (num) - C (denom) + D where is the previous estimate, and D is chosen to be large enough for example, to avoid a negative denominator - - and probably
=
large enough to exceed C (denom)
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 107
Other discriminative training criteria
There are other criteria besides MMI. For example, the MPE (minimum phone error rate) criterion due to Povey
Instead of optimizing the posterior probability of the correct hypothesis, optimizes the phone error rate See Poveys Cambridge thesis for very useful details on how to compute MPE and MMI estimates using lattices
The MCE (minimum classification error) criterion
Tries to approximate the expected word error rate
Very active research area over the past decade
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 108
15
4/13/10
Does discriminative training work?
Very impressive gains in accuracy over the MLE
20% reduction in the number of errors not uncommon * But depends on the quantity of data Small vocabulary (initial success here) Large vocabulary (success here came later with lattices)
Main idea was
Keep the model structure unchanged But try to match the means of training to the actual classification task
But heres the question:
Does the model structure need to change to make further progress? * The independence assumptions arent actually valid! * What would be a better form for the underlying model?
Could use a variant model structure as a post-processing step
Only act on an nbest list, for example, to minimize computation
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 109
A few words about simulation
How can you study the performance of an algorithm effectively? Simulate data from a known model We can run any of the algorithms weve discussed on simulated data
Is the implementation correct? How does the algorithm perform when the model is correct? How does it perform if we violate the model in a certain way?
Easy to generate random variables from an arbitrary distribution
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 110
16
4/13/10
Simulating a random variable
Suppose we wish to generate an instance of a random variable Y with cumulative distribution function (cdf) F. In other words, we want Y to be such that P(Y y) = F(y) Suppose U ~ Uniform[0,1], so U is uniform on the unit interval Let Y = F -1 (U ) Then Y has the cdf F because P(Y y) = P(F -1 (U ) y) = P(U F(y)) = F(y) Here we have relied on the assumption that F is monotone increasing
6.345/HST.728 Automatic Speech Recognition (2010)
HMMs 111
Simulating a Gaussian
If we wish to simulate a N(0,1) random variable, we just let F -1 (u) = -1 (u), which can be stored in a table. Then generate U, a uniform[0,1] random variable, and lookup Y = -1 (U ) in the table. If we wish to simulate X ~ N(, 2 ), then just let X = + Y
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 112
17
4/13/10
Simulating a discrete distribution
Suppose that we wish to simulate a random variable Y with discrete distribution p(i) : P(Y = i) = p(i) for i = 1,..., k and p(i) = 1
i
Generate U ~ Uniform[0,1] Divide up the unit interval into k regions, where the ith region R i has length p(i) If U R i , then set Y = i
6.345/HST.728 Automatic Speech Recognition (2010) HMMs 113
18